Oct 31 07:28:29.000236 ip-10-0-130-121 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 07:28:29.389509 ip-10-0-130-121 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:29.389509 ip-10-0-130-121 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 07:28:29.389509 ip-10-0-130-121 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:29.389509 ip-10-0-130-121 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:29.396462 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396373    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:29.396462 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396457    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:29.396462 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396464    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396477    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396482    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396486    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396491    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396494    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396498    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396502    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396506    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396511    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396514    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396518    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396522    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396526    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396530    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396535    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396539    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396543    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396547    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:29.396673 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396551    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396555    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396559    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396563    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396566    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396570    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396575    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396579    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396584    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396588    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396593    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396597    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396601    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396605    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396611    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396616    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396620    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396625    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396629    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:29.397436 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396634    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396638    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396642    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396647    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396651    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396656    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396660    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396665    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396669    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396673    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396677    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396682    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396686    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396691    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396695    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396699    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396703    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396708    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396714    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396718    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:29.398141 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396723    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396730    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396735    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396742    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396747    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396751    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396755    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396760    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396765    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396769    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396773    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396778    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396782    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396787    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396791    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396795    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396799    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396804    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396808    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:29.398884 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396812    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396816    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396820    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396825    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396830    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396833    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396838    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396842    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396846    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396850    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396854    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396859    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396863    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396867    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396872    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396878    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396883    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396888    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396892    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396896    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:29.399455 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396901    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396906    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396912    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396916    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396921    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396941    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.396945    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399575    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399587    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399591    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399595    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399598    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399601    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399604    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399607    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399609    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399612    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399614    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399617    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:29.399961 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399621    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399626    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399629    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399632    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399635    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399638    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399640    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399643    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399645    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399648    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399652    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399656    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399659    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399662    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399665    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399668    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399670    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399673    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:29.400409 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399675    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399678    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399681    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399684    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399687    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399690    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399693    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399695    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399698    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399700    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399703    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399706    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399708    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399711    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399713    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399716    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399719    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399721    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399724    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399726    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:29.400897 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399729    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399732    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399734    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399737    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399739    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399742    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399745    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399749    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399752    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399754    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399757    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399760    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399763    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399765    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399768    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399771    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399774    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399777    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399779    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399782    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:29.401398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399785    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399787    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399790    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399793    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399795    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399798    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399800    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399803    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399806    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399808    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399811    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399814    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399817    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399820    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399823    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399825    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399828    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399830    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399833    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399835    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:29.401914 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399838    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399840    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399843    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399845    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399848    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399851    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399854    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399857    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399859    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399863    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399866    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399869    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399871    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399874    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399877    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.399879    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400367    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400373    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400376    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400379    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:29.402406 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400382    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400384    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400387    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400390    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400393    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400396    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400399    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400402    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400404    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400407    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400410    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400413    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400415    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400418    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400421    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400423    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400426    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400428    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400432    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:29.402871 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400435    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400437    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400440    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400444    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400446    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400449    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400452    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400454    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400457    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400459    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400462    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400464    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400467    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400470    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400473    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400476    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400479    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400482    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400484    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:29.403343 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400488    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400490    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400493    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400496    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400498    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400501    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400503    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400506    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400509    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400512    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400514    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400517    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400521    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400524    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400527    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400530    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400532    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400535    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400537    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400540    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:29.403788 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400543    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400545    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400548    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400550    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400553    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400555    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400558    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400560    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400563    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400566    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400570    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400572    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400575    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400578    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400581    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400584    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400586    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400589    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400592    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400594    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:29.404300 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400597    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400599    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400602    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400604    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400607    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400610    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400613    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400616    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400618    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400621    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400624    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400626    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400629    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400632    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400634    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400638    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400641    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400643    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400646    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:29.404753 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400648    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400651    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400653    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400656    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.400658    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402447    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402459    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402467    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402472    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402477    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402481    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402485    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402490    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402493    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402496    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402500    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402503    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402507    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402510    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402513    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402516    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402519    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402523    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 07:28:29.405212 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402527    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402530    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402533    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402536    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402539    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402543    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402546    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402550    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402553    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402556    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402559    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402562    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402565    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402568    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402573    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402577    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402580    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402583    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402587    2572 flags.go:64] FLAG: --enable-server="true"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402590    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402612    2572 flags.go:64] FLAG: --event-burst="100"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402615    2572 flags.go:64] FLAG: --event-qps="50"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402618    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402621    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 07:28:29.405734 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402625    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402637    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402640    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402644    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402647    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402650    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402653    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402656    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402659    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402662    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402665    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402668    2572 flags.go:64] FLAG: --feature-gates=""
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402672    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402675    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402679    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402682    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402685    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402688    2572 flags.go:64] FLAG: --help="false"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402691    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402694    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402698    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402701    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402704    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 07:28:29.406323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402708    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402711    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402714    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402718    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402721    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402725    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402728    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402731    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402734    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402737    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402740    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402744    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402747    2572 flags.go:64] FLAG: --lock-file=""
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402750    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402753    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402756    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402762    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402765    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402768    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402771    2572 flags.go:64] FLAG: --logging-format="text"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402774    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402777    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402780    2572 flags.go:64] FLAG: --manifest-url=""
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402783    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402788    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 07:28:29.406871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402791    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402795    2572 flags.go:64] FLAG: --max-pods="110"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402798    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402801    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402804    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402807    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402810    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402813    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402816    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402823    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402826    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402830    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402833    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402836    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402839    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402842    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402845    2572 flags.go:64] FLAG: --port="10250"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402848    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402851    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-022935f6e06f361fa"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402855    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402857    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402860    2572 flags.go:64] FLAG: --register-node="true"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402863    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402868    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402871    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402874    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 07:28:29.407586 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402877    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402881    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402884    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402887    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402890    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402893    2572 flags.go:64] FLAG: --runonce="false"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402896    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402900    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402903    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402905    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402908    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402911    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402915    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402918    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402920    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402933    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402937    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402940    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402944    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402948    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402950    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402956    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402958    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402961    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402965    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 31 07:28:29.408219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402968    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402971    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402974    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402977    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402980    2572 flags.go:64] FLAG: --v="2"
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402984    2572 flags.go:64] FLAG: --version="false"
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402988    2572 flags.go:64] FLAG: --vmodule=""
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402992    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.402995    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403151    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403155    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403159    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403162    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403165    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403168    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403170    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403173    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403176    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403178    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403181    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403185    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:29.408808 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403188    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403191    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403194    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403198    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403202    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403205    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403208    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403212    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403214    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403217    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403220    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403223    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403225    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403228    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403230    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403233    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403236    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403239    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403242    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:29.409318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403244    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403247    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403249    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403252    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403255    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403257    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403260    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403263    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403265    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403268    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403271    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403273    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403276    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403278    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403281    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403283    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403286    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403289    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403291    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:29.409773 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403296    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403299    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403302    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403305    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403308    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403311    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403314    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403317    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403319    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403322    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403325    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403327    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403330    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403333    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403335    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403338    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403340    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403343    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403346    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403348    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:29.410270 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403351    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403354    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403357    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403359    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403362    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403367    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403370    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403372    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403375    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403377    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403380    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403383    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403385    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403389    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403394    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403396    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403399    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403402    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403405    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:29.410775 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403408    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403410    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403413    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403416    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403419    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403421    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403424    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403427    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403430    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403433    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403435    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403438    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403441    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403444    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403446    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403449    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:29.411250 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403451    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.403477    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403514    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403518    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403522    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403524    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403527    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403530    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403533    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403536    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403538    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403541    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403546    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403549    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403551    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403554    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:29.411623 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403557    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403560    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403563    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403565    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403568    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403571    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403574    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403577    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403579    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403582    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403585    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403587    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403590    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403593    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403595    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403598    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403601    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403603    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403607    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:29.412022 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403610    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403612    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403615    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403618    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403620    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403623    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403626    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403628    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403631    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403636    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403638    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403641    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403643    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403646    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403649    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403652    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403654    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403657    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403659    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403663    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:29.412463 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403665    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403668    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403670    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403673    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403676    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403678    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403681    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403685    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403689    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403692    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403695    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403699    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403701    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403704    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403706    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403709    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403711    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403714    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403716    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:29.412940 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403719    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403722    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403726    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403730    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403733    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403735    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403738    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403741    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403743    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403746    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403749    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403752    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403754    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403757    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403760    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403762    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403765    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403768    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403770    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403773    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:29.413398 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403776    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403778    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403781    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403785    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403790    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403793    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403796    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403799    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403801    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403804    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403807    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403809    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403814    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.403816    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.412049    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.412065    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.412081    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.412087    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412143    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:29.413861 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412147    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412151    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412154    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412157    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412160    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412163    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412165    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412168    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412171    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412174    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412176    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412179    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412182    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412184    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412187    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412190    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412192    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412195    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412198    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412201    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:29.414318 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412204    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412206    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412209    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412212    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412215    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412217    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412220    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412223    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412226    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412230    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412233    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412236    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412239    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412242    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412245    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412247    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412250    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412252    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412255    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:29.414783 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412258    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412261    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412264    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412268    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412271    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412274    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412277    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412280    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412283    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412286    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412288    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412291    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412294    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412296    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412299    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412302    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412305    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412308    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412310    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:29.415357 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412313    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412316    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412320    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412322    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412326    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412329    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412331    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412334    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412337    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412339    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412342    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412344    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412347    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412350    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412353    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412356    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412358    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412361    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412364    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:29.415800 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412366    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412369    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412371    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412374    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412377    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412380    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412384    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412387    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412390    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412393    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412396    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412398    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412401    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412404    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412406    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412409    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412412    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412415    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412418    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412421    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:29.416288 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412424    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412426    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412429    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412431    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412434    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412436    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412439    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412442    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.412470    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412629    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412634    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412637    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412641    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412644    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412646    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412649    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:29.416743 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412652    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412654    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412657    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412661    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412665    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412668    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412671    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412674    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412677    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412679    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412682    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412685    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412687    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412690    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412693    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412697    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412699    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412702    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412705    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:29.417133 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412708    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412711    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412713    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412716    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412719    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412721    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412724    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412726    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412729    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412732    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412734    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412737    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412739    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412742    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412744    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412747    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412749    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412752    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412754    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:29.417581 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412757    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412759    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412762    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412765    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412767    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412782    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412785    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412789    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412792    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412795    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412798    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412801    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412803    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412806    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412809    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412812    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412814    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412817    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412819    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412822    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:29.418030 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412825    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412827    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412830    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412832    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412835    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412837    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412840    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412842    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412845    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412848    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412850    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412853    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412855    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412858    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412861    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412865    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412868    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412871    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412874    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:29.418498 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412877    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412879    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412882    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412885    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412887    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412890    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412893    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412896    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412898    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412901    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412903    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412906    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412908    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412911    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412914    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412916    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412919    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412922    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412936    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:29.418938 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412939    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412941    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.412944    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.412969    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413003    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413007    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413010    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413012    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413015    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413018    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413020    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413023    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413026    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413028    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413031    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413034    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:29.419373 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413036    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413039    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413042    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413045    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413047    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413050    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413053    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413055    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413058    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413060    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413063    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413065    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413068    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413071    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413073    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413075    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413078    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413081    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413083    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413086    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:29.419752 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413088    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413091    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413093    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413096    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413098    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413101    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413105    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413110    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413113    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413116    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413119    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413122    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413125    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413128    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413130    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413133    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413136    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413139    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413141    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:29.420219 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413144    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413147    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413149    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413151    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413154    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413157    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413159    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413162    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413172    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413175    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413178    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413181    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413183    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413186    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413188    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413191    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413193    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413196    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413199    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:29.420662 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413201    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413207    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413209    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413212    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413217    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413221    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413224    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413227    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413230    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413233    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413236    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413239    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413242    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413245    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413248    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413250    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413253    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413255    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413258    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:29.421108 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413260    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413263    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413266    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413271    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413273    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413276    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413278    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413281    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413284    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413287    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413290    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413293    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413296    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413298    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413302    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413304    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:29.413307    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.413358    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.415661    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 07:28:29.422243 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.416766    2572 server.go:1014] "Starting client certificate rotation"
Oct 31 07:28:29.422998 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.416867    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:28:29.422998 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.416898    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:28:29.422998 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.419023    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:28:29.422998 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.420179    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:28:29.437849 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.437831    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 31 07:28:29.442481 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.442467    2572 log.go:25] "Validated CRI v1 image API"
Oct 31 07:28:29.443322 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.443299    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 07:28:29.447441 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.447422    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f3ffebdc-4b6e-4983-ad76-d443fe514e70:/dev/nvme0n1p4 fe4880a2-a519-45fa-81f8-3258dbd2f6f8:/dev/nvme0n1p3]
Oct 31 07:28:29.447501 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.447440    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 07:28:29.449233 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.449202    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 07:28:29.452817 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.452710    2572 manager.go:217] Machine: {Timestamp:2026-10-31 07:28:29.4512398 +0000 UTC m=+0.354841862 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101182 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec219188a6970aae52828606e75c02c7 SystemUUID:ec219188-a697-0aae-5282-8606e75c02c7 BootID:cfefab99-c1c9-4fe0-862a-d5fe14dcd440 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d7:0d:b2:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d7:0d:b2:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:6a:e1:79:30:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 07:28:29.452817 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.452814    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 07:28:29.452943 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.452919    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 07:28:29.456561 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.456535    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 07:28:29.456690 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.456563    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-121.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 07:28:29.456754 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.456692    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 07:28:29.456754 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.456700    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 07:28:29.456754 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.456745    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:28:29.456836 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.456759    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:28:29.458024 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.458013    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 07:28:29.462697 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.462686    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 07:28:29.462828 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.462819    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 07:28:29.462869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.462850    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 07:28:29.462869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.462861    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 07:28:29.462869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.462870    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 07:28:29.463023 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.462878    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 31 07:28:29.463023 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.462887    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 07:28:29.463221 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.463200    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:28:29.465695 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.465664    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 07:28:29.473877 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.473861    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:28:29.474834 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.474820    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 07:28:29.474877 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.474861    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 07:28:29.475044 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475034    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 07:28:29.475075 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475050    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 07:28:29.475075 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475057    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 07:28:29.475075 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475062    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 07:28:29.475075 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475068    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 07:28:29.475075 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475074    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 07:28:29.475831 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475820    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 07:28:29.475865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475836    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 07:28:29.475865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475851    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 07:28:29.475865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475862    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 07:28:29.476059 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475875    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 07:28:29.476059 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.475888    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 07:28:29.477327 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.477314    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 07:28:29.477360 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.477330    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 07:28:29.482402 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.482388    2572 server.go:1263] "Started kubelet"
Oct 31 07:28:29.482520 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.482473    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 07:28:29.482576 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.482540    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 07:28:29.482628 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.482497    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 07:28:29.483234 ip-10-0-130-121 systemd[1]: Started Kubernetes Kubelet.
Oct 31 07:28:29.484755 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.484740    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 07:28:29.484838 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.484753    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r9rz9"
Oct 31 07:28:29.485509 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.485490    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 07:28:29.488026 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.487644    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 07:28:29.488026 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.487653    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-121.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 07:28:29.488026 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.487654    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-121.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 07:28:29.489457 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.489436    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 07:28:29.491686 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.491670    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r9rz9"
Oct 31 07:28:29.493963 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.493947    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 07:28:29.495258 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.495243    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 07:28:29.495368 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.495337    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:29.496068 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496037    2572 factory.go:55] Registering systemd factory
Oct 31 07:28:29.496068 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496058    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 31 07:28:29.496068 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496069    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 07:28:29.496303 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496071    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 07:28:29.496303 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496089    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 07:28:29.496303 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496227    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 07:28:29.496303 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496236    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 07:28:29.496303 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496266    2572 factory.go:153] Registering CRI-O factory
Oct 31 07:28:29.496303 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496278    2572 factory.go:223] Registration of the crio container factory successfully
Oct 31 07:28:29.496574 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496319    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 07:28:29.496574 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496334    2572 factory.go:103] Registering Raw factory
Oct 31 07:28:29.496574 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.496343    2572 manager.go:1199] Started watching for new ooms in manager
Oct 31 07:28:29.497316 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.497299    2572 manager.go:319] Starting recovery of all containers
Oct 31 07:28:29.502442 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.502419    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:29.505781 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.505754    2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-121.us-east-2.compute.internal\" not found" node="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.510952 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.510915    2572 manager.go:324] Recovery completed
Oct 31 07:28:29.515300 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.515285    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:29.517170 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.517149    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:29.517232 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.517185    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:29.517232 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.517196    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:29.517794 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.517780    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 31 07:28:29.517794 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.517791    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 07:28:29.517912 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.517803    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:29.517912 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.517806    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 07:28:29.518473 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.518459    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:29.518525 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.518482    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:29.518525 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.518491    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:29.520044 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.520031    2572 policy_none.go:50] "Start"
Oct 31 07:28:29.520108 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.520049    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 07:28:29.520108 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.520066    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 07:28:29.522158 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.522146    2572 policy_none.go:44] "Start"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.568257    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.573593    2572 manager.go:341] "Starting Device Plugin manager"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.573619    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.573628    2572 server.go:87] "Starting device plugin registration server"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.573898    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.573911    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.573987    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.574052    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.574059    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.574650    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.574703    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.577507    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.578620    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.578633    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.578650    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 07:28:29.581367 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.578682    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 07:28:29.583829 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.582073    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:29.674575 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.674517    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:29.675465 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.675446    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:29.675559 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.675483    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:29.675559 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.675496    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:29.675559 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.675526    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.678821 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.678804    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"]
Oct 31 07:28:29.678871 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.678860    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:29.679617 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.679598    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:29.679663 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.679636    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:29.679663 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.679651    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:29.683005 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.682991    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:29.683648 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.683634    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:29.683710 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.683658    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:29.683710 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.683668    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:29.683812 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.683787    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.683859 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.683825    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:29.684477 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.684461    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:29.684543 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.684488    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:29.684543 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.684500    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:29.685700 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.685685    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.685767 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.685710    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:29.686366 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.686350    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.686430 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.686382    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-121.us-east-2.compute.internal\": node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:29.686430 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.686350    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:29.686515 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.686452    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:29.686515 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.686463    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:29.712533 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.712515    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found" node="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.713309 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.713294    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:29.715604 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.715590    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found" node="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.797226 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.797197    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.797226 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.797226    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.797379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.797247    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a42df5a3903764c9a9afa19fe2c78097-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"a42df5a3903764c9a9afa19fe2c78097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.813994 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.813968    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:29.898201 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.898159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.898372 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.898206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.898372 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.898268    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a42df5a3903764c9a9afa19fe2c78097-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"a42df5a3903764c9a9afa19fe2c78097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.898372 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.898282    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.898372 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.898315    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a42df5a3903764c9a9afa19fe2c78097-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"a42df5a3903764c9a9afa19fe2c78097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.898372 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:29.898324    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:29.914239 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:29.914220    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:30.015074 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:30.015050    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:30.018241 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:30.018221    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:30.020441 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:30.020406    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:30.116293 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:30.116115    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:31.396881 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:31.396850    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:31.497319 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:31.497241    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:31.497880 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:31.497860    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:31.596419 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:31.596393    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 07:28:31.596884 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:31.596564    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:31.596884 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:31.596571    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:31.596884 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:31.596585    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:31.597469 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:31.597453    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:31.675266 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:31.675218    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:23:30 +0000 UTC" deadline="2028-06-01 01:54:04.371944749 +0000 UTC"
Oct 31 07:28:31.675266 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:31.675260    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13890h25m32.696686736s"
Oct 31 07:28:31.703084 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:31.703057    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:31.804068 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:31.804004    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:31.845584 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:31.845552    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51efa451cfc3022379daa87dd3ede24.slice/crio-b14dfce64a90aa78f790e735f4a689603802b3ee22d0964bc8702bdd43c86beb WatchSource:0}: Error finding container b14dfce64a90aa78f790e735f4a689603802b3ee22d0964bc8702bdd43c86beb: Status 404 returned error can't find the container with id b14dfce64a90aa78f790e735f4a689603802b3ee22d0964bc8702bdd43c86beb
Oct 31 07:28:31.851268 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:31.851251    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:28:31.888962 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:31.888937    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42df5a3903764c9a9afa19fe2c78097.slice/crio-7fe1b3d7887719f095609272bea432bd413aef0b6ca9b72f0b3c1a8b8b2811b4 WatchSource:0}: Error finding container 7fe1b3d7887719f095609272bea432bd413aef0b6ca9b72f0b3c1a8b8b2811b4: Status 404 returned error can't find the container with id 7fe1b3d7887719f095609272bea432bd413aef0b6ca9b72f0b3c1a8b8b2811b4
Oct 31 07:28:31.904544 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:31.904520    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:32.005046 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.005011    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-121.us-east-2.compute.internal\" not found"
Oct 31 07:28:32.040468 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.040447    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:32.076234 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.076146    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:32.089543 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.089525    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:32.090660 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.090648    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:28:32.098859 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.098844    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:32.572960 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.572924    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:32.644159 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.643816    2572 apiserver.go:52] "Watching apiserver"
Oct 31 07:28:32.653099 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.653068    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 07:28:32.654082 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.654048    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-d5t72","kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-d4l58","openshift-network-operator/iptables-alerter-6859w","kube-system/global-pull-secret-syncer-l4tmf","kube-system/konnectivity-agent-sbln7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w","openshift-dns/node-resolver-49r2r","openshift-ovn-kubernetes/ovnkube-node-vvnff","openshift-cluster-node-tuning-operator/tuned-nrtcz","openshift-image-registry/node-ca-fz4tp","openshift-multus/multus-hmj2s","openshift-multus/network-metrics-daemon-svhj9"]
Oct 31 07:28:32.657147 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.657128    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.659548 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.659529    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.661708 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.661548    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:32.661708 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.661659    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:32.661881 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.661733    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:32.661881 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.661867    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 07:28:32.662006 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.661947    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k6p7p\""
Oct 31 07:28:32.662872 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.662847    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 07:28:32.662978 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.662909    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 07:28:32.662978 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.662910    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x8xx7\""
Oct 31 07:28:32.663885 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.663867    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:32.663977 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.663934    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:32.664044 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.663976    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 07:28:32.664044 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.664032    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 07:28:32.664873 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.664853    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ghqlz\""
Oct 31 07:28:32.664958 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.664892    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 07:28:32.664958 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.664902    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 07:28:32.667313 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.667291    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.669641 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.669619    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.669772 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.669691    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:32.671719 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.671695    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.674517 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.674491    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.674967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.674946    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 07:28:32.678436 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.678417    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.679548 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.679508    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.682799 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.682780    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.686840 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.686820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:32.686932 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.686884    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:32.686932 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.686922    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 07:28:32.687480 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687445    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:32.687568 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687494    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-systemd-units\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.687568 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-system-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.687671 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-hostroot\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.687671 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687642    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-kubernetes\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.687778 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687678    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-multus\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.687831 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-device-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.687883 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687833    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-node-log\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.687883 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.688032 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687890    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-tuned\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.688032 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687914    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-bin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.688032 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.687936    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-etc-kubernetes\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.688173 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688031    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.688173 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688059    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-log-socket\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.688173 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-var-lib-kubelet\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.688173 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688113    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fxd\" (UniqueName: \"kubernetes.io/projected/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-kube-api-access-m9fxd\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.688173 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688139    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-socket-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.688173 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688162    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3d13df-dc50-41c8-a7cd-051fb9934a34-tmp-dir\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688202    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-systemd\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-conf-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688321    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688345    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-sys-fs\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b3d13df-dc50-41c8-a7cd-051fb9934a34-hosts-file\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-netd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688416    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-host-slash\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrfw\" (UniqueName: \"kubernetes.io/projected/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-kube-api-access-rjrfw\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.688479 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-multus-certs\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688526    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-iptables-alerter-script\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688556    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc9xp\" (UniqueName: \"kubernetes.io/projected/5b3d13df-dc50-41c8-a7cd-051fb9934a34-kube-api-access-pc9xp\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688605    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-etc-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-env-overrides\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688672    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-modprobe-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688695    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-sys\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688718    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-lib-modules\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688742    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-bin\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-script-lib\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-conf\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-socket-dir-parent\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688898    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.688946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688923    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xsdc\" (UniqueName: \"kubernetes.io/projected/a329ba42-e8f3-43ed-9355-d906a577e176-kube-api-access-5xsdc\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688962    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.688988    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689085    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-slash\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cni-binary-copy\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689135    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6121aa17-a37d-47de-af76-08c2aa174568-agent-certs\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689163    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-kubelet-config\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-config\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689228    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/735d44d3-7a63-4a36-8278-452074cec97c-ovn-node-metrics-cert\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689252    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-netns\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689330    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-daemon-config\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689357    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-system-cni-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689381    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-kubelet\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689417    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-netns\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysconfig\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689473    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-k8s-cni-cncf-io\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.689611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689498    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-registration-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689568    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-systemd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689593    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-ovn\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689615    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-run\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689639    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cnibin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689663    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-kubelet\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-dbus\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689713    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgww\" (UniqueName: \"kubernetes.io/projected/021b18d4-0686-43c1-9faf-b1ca555d22a3-kube-api-access-6vgww\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689738    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-cnibin\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689763    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc79k\" (UniqueName: \"kubernetes.io/projected/0546b914-d103-41a5-9e5b-13c6e41bcd85-kube-api-access-xc79k\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljlb\" (UniqueName: \"kubernetes.io/projected/735d44d3-7a63-4a36-8278-452074cec97c-kube-api-access-mljlb\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689867    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-host\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689917    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-var-lib-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.690396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.689998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.691166 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.690029    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-tmp\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.691166 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.690121    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6121aa17-a37d-47de-af76-08c2aa174568-konnectivity-ca\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:32.700456 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.700429    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r8hk8"
Oct 31 07:28:32.707066 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.707044    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r8hk8"
Oct 31 07:28:32.762119 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.762085    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nq6dm\""
Oct 31 07:28:32.762309 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.762291    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bn5lf\""
Oct 31 07:28:32.762575 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.762557    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 07:28:32.762678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.762606    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 07:28:32.762738 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.762713    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 07:28:32.762785 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.762757    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 07:28:32.762785 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.762770    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7sl8v\""
Oct 31 07:28:32.763483 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.763455    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 07:28:32.763798 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.763775    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9r8qc\""
Oct 31 07:28:32.763890 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.763832    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zjbzq\""
Oct 31 07:28:32.763890 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.763855    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6w9r4\""
Oct 31 07:28:32.763992 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.763897    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:32.763992 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.763946    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 07:28:32.764407 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764388    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 07:28:32.764506 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764442    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:32.764506 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764479    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 07:28:32.764506 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764483    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 07:28:32.764647 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764552    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 07:28:32.764647 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764588    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 07:28:32.764647 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764595    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 07:28:32.764647 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764640    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 07:28:32.764811 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764388    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 07:28:32.764811 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.764673    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 07:28:32.765232 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.765161    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"a42df5a3903764c9a9afa19fe2c78097","Type":"ContainerStarted","Data":"7fe1b3d7887719f095609272bea432bd413aef0b6ca9b72f0b3c1a8b8b2811b4"}
Oct 31 07:28:32.766104 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.766085    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 07:28:32.766452 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.766391    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"d51efa451cfc3022379daa87dd3ede24","Type":"ContainerStarted","Data":"b14dfce64a90aa78f790e735f4a689603802b3ee22d0964bc8702bdd43c86beb"}
Oct 31 07:28:32.777583 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.777556    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 07:28:32.790980 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.790945    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-kubelet\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.790980 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.790988    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-dbus\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.791218 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgww\" (UniqueName: \"kubernetes.io/projected/021b18d4-0686-43c1-9faf-b1ca555d22a3-kube-api-access-6vgww\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.791218 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791048    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-cnibin\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.791218 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791061    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-kubelet\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.791218 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791073    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc79k\" (UniqueName: \"kubernetes.io/projected/0546b914-d103-41a5-9e5b-13c6e41bcd85-kube-api-access-xc79k\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.791218 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791108    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljlb\" (UniqueName: \"kubernetes.io/projected/735d44d3-7a63-4a36-8278-452074cec97c-kube-api-access-mljlb\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.791218 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791132    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-host\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.791218 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791160    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6bn\" (UniqueName: \"kubernetes.io/projected/b5bca13e-12d2-4673-8223-92f7755eac90-kube-api-access-zl6bn\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:32.791218 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-var-lib-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791233    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-cnibin\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791258    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-host\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791285    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791197    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-dbus\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791346    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-var-lib-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791374    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791399    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-tmp\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791423    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6121aa17-a37d-47de-af76-08c2aa174568-konnectivity-ca\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791474    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-systemd-units\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-system-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791523    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-hostroot\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791550    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4acf6e61-938d-495e-92ea-c1bc343b9406-host\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.791602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791555    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791576    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-kubernetes\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791601    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-multus\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791613    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-systemd-units\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791625    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-device-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791650    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-node-log\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791675    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791703    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-tuned\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791727    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-bin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791750    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-etc-kubernetes\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791754    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-kubernetes\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791762    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-multus\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4acf6e61-938d-495e-92ea-c1bc343b9406-serviceca\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791803    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791826    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791810    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-system-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791872    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-node-log\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791899    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-device-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.792317 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791920    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-etc-kubernetes\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791935    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-hostroot\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791937    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.791958    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792005    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-bin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792008    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-log-socket\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-var-lib-kubelet\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792113    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-log-socket\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792197    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-var-lib-kubelet\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fxd\" (UniqueName: \"kubernetes.io/projected/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-kube-api-access-m9fxd\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792288    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-socket-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792313    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3d13df-dc50-41c8-a7cd-051fb9934a34-tmp-dir\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792336    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-systemd\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-conf-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792414    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792436    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-socket-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792438    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 07:28:32.793037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792439    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-sys-fs\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b3d13df-dc50-41c8-a7cd-051fb9934a34-hosts-file\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792492    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-sys-fs\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-netd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792532    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-host-slash\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrfw\" (UniqueName: \"kubernetes.io/projected/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-kube-api-access-rjrfw\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-multus-certs\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792607    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-iptables-alerter-script\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792655    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9xp\" (UniqueName: \"kubernetes.io/projected/5b3d13df-dc50-41c8-a7cd-051fb9934a34-kube-api-access-pc9xp\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792678    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-etc-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792704    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-env-overrides\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792728    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-modprobe-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-sys\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792757    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-systemd\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792776    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-lib-modules\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-bin\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.793865 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792803    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3d13df-dc50-41c8-a7cd-051fb9934a34-tmp-dir\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792814    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792876    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-conf-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792881    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792934    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792978    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-host-slash\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793022    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-etc-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793038    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b3d13df-dc50-41c8-a7cd-051fb9934a34-hosts-file\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793076    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-multus-certs\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793246    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-netd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792824    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-script-lib\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793289    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-conf\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793315    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-socket-dir-parent\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793353    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-modprobe-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsdc\" (UniqueName: \"kubernetes.io/projected/a329ba42-e8f3-43ed-9355-d906a577e176-kube-api-access-5xsdc\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793395    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5nn6\" (UniqueName: \"kubernetes.io/projected/4acf6e61-938d-495e-92ea-c1bc343b9406-kube-api-access-r5nn6\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.794802 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793427    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793431    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793469    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793496    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-slash\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793524    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cni-binary-copy\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793548    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6121aa17-a37d-47de-af76-08c2aa174568-agent-certs\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-kubelet-config\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793581    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-env-overrides\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793601    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-config\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793629    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/735d44d3-7a63-4a36-8278-452074cec97c-ovn-node-metrics-cert\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793657    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-sys\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793672    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-netns\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793699    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-daemon-config\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793736    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-system-cni-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793741    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6121aa17-a37d-47de-af76-08c2aa174568-konnectivity-ca\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793761    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-kubelet\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793789    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-netns\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.795550 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793805    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-lib-modules\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793815    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysconfig\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793842    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-k8s-cni-cncf-io\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793846    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-slash\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793863    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793874    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-registration-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793884    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-bin\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793806    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-kubelet-config\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793903    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-systemd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793929    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-ovn\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793960    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-run\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793981    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cnibin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.793994    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-conf\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.792704    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794031    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysconfig\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794063    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-systemd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794073    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-ovn\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.793983    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:32.796238 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794103    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-k8s-cni-cncf-io\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794159    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-run\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794221    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cnibin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.794299    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret podName:9420864f-c0f7-47d3-ad36-b4b15ba44b57 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:33.294278408 +0000 UTC m=+3.017623752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret") pod "global-pull-secret-syncer-l4tmf" (UID: "9420864f-c0f7-47d3-ad36-b4b15ba44b57") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794286    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cni-binary-copy\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794384    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794543    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-daemon-config\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794569    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-config\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794620    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-registration-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794755    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-system-cni-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794791    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-kubelet\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.794825    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-netns\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.795154    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-netns\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.795212    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-socket-dir-parent\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.795933    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-iptables-alerter-script\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.796906 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.795981    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-script-lib\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.798350 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.798329    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:32.798350 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.798350    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:32.798525 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.798362    2572 projected.go:196] Error preparing data for projected volume kube-api-access-j988n for pod openshift-network-diagnostics/network-check-target-d5t72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:32.798525 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.798411    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n podName:686827ed-5eb8-4cdc-aeca-176b015eb4f8 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:33.298394542 +0000 UTC m=+3.021739880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j988n" (UniqueName: "kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n") pod "network-check-target-d5t72" (UID: "686827ed-5eb8-4cdc-aeca-176b015eb4f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:32.798660 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.798608    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-tmp\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.798969 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.798946    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6121aa17-a37d-47de-af76-08c2aa174568-agent-certs\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:32.799232 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.799149    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/735d44d3-7a63-4a36-8278-452074cec97c-ovn-node-metrics-cert\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.800713 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.799894    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-tuned\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.800713 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.800681    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc79k\" (UniqueName: \"kubernetes.io/projected/0546b914-d103-41a5-9e5b-13c6e41bcd85-kube-api-access-xc79k\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:32.802530 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.802501    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrfw\" (UniqueName: \"kubernetes.io/projected/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-kube-api-access-rjrfw\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.802946 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.802911    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9xp\" (UniqueName: \"kubernetes.io/projected/5b3d13df-dc50-41c8-a7cd-051fb9934a34-kube-api-access-pc9xp\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:32.803322 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.803268    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljlb\" (UniqueName: \"kubernetes.io/projected/735d44d3-7a63-4a36-8278-452074cec97c-kube-api-access-mljlb\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:32.804602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.803550    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fxd\" (UniqueName: \"kubernetes.io/projected/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-kube-api-access-m9fxd\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.804602 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.804558    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgww\" (UniqueName: \"kubernetes.io/projected/021b18d4-0686-43c1-9faf-b1ca555d22a3-kube-api-access-6vgww\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:32.805240 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.805221    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsdc\" (UniqueName: \"kubernetes.io/projected/a329ba42-e8f3-43ed-9355-d906a577e176-kube-api-access-5xsdc\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:32.894643 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.894552    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5nn6\" (UniqueName: \"kubernetes.io/projected/4acf6e61-938d-495e-92ea-c1bc343b9406-kube-api-access-r5nn6\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.894643 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.894632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6bn\" (UniqueName: \"kubernetes.io/projected/b5bca13e-12d2-4673-8223-92f7755eac90-kube-api-access-zl6bn\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:32.894861 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.894676    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4acf6e61-938d-495e-92ea-c1bc343b9406-host\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.894861 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.894705    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4acf6e61-938d-495e-92ea-c1bc343b9406-serviceca\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.894861 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.894769    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4acf6e61-938d-495e-92ea-c1bc343b9406-host\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.894861 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.894804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:32.895027 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.894926    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:32.895027 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:32.894989    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:33.394970469 +0000 UTC m=+3.118315796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:32.895386 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.895360    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4acf6e61-938d-495e-92ea-c1bc343b9406-serviceca\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.903446 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.903405    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6bn\" (UniqueName: \"kubernetes.io/projected/b5bca13e-12d2-4673-8223-92f7755eac90-kube-api-access-zl6bn\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:32.906554 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.906526    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5nn6\" (UniqueName: \"kubernetes.io/projected/4acf6e61-938d-495e-92ea-c1bc343b9406-kube-api-access-r5nn6\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:32.974401 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.974361    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:28:32.982823 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.982801    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hmj2s"
Oct 31 07:28:32.985707 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:32.985674    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3432e0f3_ae43_4815_a8b1_38e7b7c0222d.slice/crio-0a219d65cff1ea71f8e7078396466c39bf362b51439e959c770cc7e98adf00ec WatchSource:0}: Error finding container 0a219d65cff1ea71f8e7078396466c39bf362b51439e959c770cc7e98adf00ec: Status 404 returned error can't find the container with id 0a219d65cff1ea71f8e7078396466c39bf362b51439e959c770cc7e98adf00ec
Oct 31 07:28:32.991998 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:32.991957    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8e87b5_3eda_4fd4_9b70_f72fe97084ca.slice/crio-e3e2b63ff27a4d5fab22a0ae75d10d10781aa0169521f6f236ce0748ef466767 WatchSource:0}: Error finding container e3e2b63ff27a4d5fab22a0ae75d10d10781aa0169521f6f236ce0748ef466767: Status 404 returned error can't find the container with id e3e2b63ff27a4d5fab22a0ae75d10d10781aa0169521f6f236ce0748ef466767
Oct 31 07:28:32.993035 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:32.993016    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:33.003265 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:33.003240    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6121aa17_a37d_47de_af76_08c2aa174568.slice/crio-62fb044edc2c1ec66daa935b2fadcf4f37117b386ba45dc02767f6c5fa474cd1 WatchSource:0}: Error finding container 62fb044edc2c1ec66daa935b2fadcf4f37117b386ba45dc02767f6c5fa474cd1: Status 404 returned error can't find the container with id 62fb044edc2c1ec66daa935b2fadcf4f37117b386ba45dc02767f6c5fa474cd1
Oct 31 07:28:33.019461 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.019433    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:33.022538 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.022506    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:28:33.027533 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.027512    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:28:33.029661 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.029638    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:28:33.034707 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:33.034677    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda329ba42_e8f3_43ed_9355_d906a577e176.slice/crio-064d02052d83429e3515ed4ca913ccbd9fa7652b571971250a6bd9b68cb83930 WatchSource:0}: Error finding container 064d02052d83429e3515ed4ca913ccbd9fa7652b571971250a6bd9b68cb83930: Status 404 returned error can't find the container with id 064d02052d83429e3515ed4ca913ccbd9fa7652b571971250a6bd9b68cb83930
Oct 31 07:28:33.039544 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:33.039518    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0546b914_d103_41a5_9e5b_13c6e41bcd85.slice/crio-2e27b0d544e40ec8f0656b7c9c8c64a5d7178ffcd9dd6cbc0528e661514171bc WatchSource:0}: Error finding container 2e27b0d544e40ec8f0656b7c9c8c64a5d7178ffcd9dd6cbc0528e661514171bc: Status 404 returned error can't find the container with id 2e27b0d544e40ec8f0656b7c9c8c64a5d7178ffcd9dd6cbc0528e661514171bc
Oct 31 07:28:33.040415 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:33.040390    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3d13df_dc50_41c8_a7cd_051fb9934a34.slice/crio-2f9822d2e207c346eccf487923220859228defbdb9a2878a100a2071ca3c7e18 WatchSource:0}: Error finding container 2f9822d2e207c346eccf487923220859228defbdb9a2878a100a2071ca3c7e18: Status 404 returned error can't find the container with id 2f9822d2e207c346eccf487923220859228defbdb9a2878a100a2071ca3c7e18
Oct 31 07:28:33.047952 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.047929    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:28:33.050094 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.050056    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:28:33.054281 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.054127    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:33.060701 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:33.060669    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021b18d4_0686_43c1_9faf_b1ca555d22a3.slice/crio-827873cf42285fdb51dcb79e7d1a31808677f60b5d944e71ed938f93df9a0859 WatchSource:0}: Error finding container 827873cf42285fdb51dcb79e7d1a31808677f60b5d944e71ed938f93df9a0859: Status 404 returned error can't find the container with id 827873cf42285fdb51dcb79e7d1a31808677f60b5d944e71ed938f93df9a0859
Oct 31 07:28:33.068024 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:33.067814    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acf6e61_938d_495e_92ea_c1bc343b9406.slice/crio-4ae2672ff707f35394a0a8bfb496ed32c358570deaf62300c91b68bcbaba8863 WatchSource:0}: Error finding container 4ae2672ff707f35394a0a8bfb496ed32c358570deaf62300c91b68bcbaba8863: Status 404 returned error can't find the container with id 4ae2672ff707f35394a0a8bfb496ed32c358570deaf62300c91b68bcbaba8863
Oct 31 07:28:33.069856 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:28:33.069656    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735d44d3_7a63_4a36_8278_452074cec97c.slice/crio-e5b849564cdfc431c9a4da01bb4b99d3c869a00c462b30f2177a77cef3636660 WatchSource:0}: Error finding container e5b849564cdfc431c9a4da01bb4b99d3c869a00c462b30f2177a77cef3636660: Status 404 returned error can't find the container with id e5b849564cdfc431c9a4da01bb4b99d3c869a00c462b30f2177a77cef3636660
Oct 31 07:28:33.297863 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.297828    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:33.298012 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:33.297939    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:33.298012 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:33.297997    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret podName:9420864f-c0f7-47d3-ad36-b4b15ba44b57 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:34.29798096 +0000 UTC m=+4.021326284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret") pod "global-pull-secret-syncer-l4tmf" (UID: "9420864f-c0f7-47d3-ad36-b4b15ba44b57") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:33.398500 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.398460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:33.398661 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.398577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:33.398661 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:33.398623    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:33.398661 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:33.398647    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:33.398661 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:33.398663    2572 projected.go:196] Error preparing data for projected volume kube-api-access-j988n for pod openshift-network-diagnostics/network-check-target-d5t72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:33.398861 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:33.398696    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:33.398861 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:33.398729    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n podName:686827ed-5eb8-4cdc-aeca-176b015eb4f8 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:34.398711388 +0000 UTC m=+4.122056715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j988n" (UniqueName: "kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n") pod "network-check-target-d5t72" (UID: "686827ed-5eb8-4cdc-aeca-176b015eb4f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:33.398861 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:33.398758    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:34.39874765 +0000 UTC m=+4.122092974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:33.660383 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.660280    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:33.711310 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.711179    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:32 +0000 UTC" deadline="2028-08-09 16:59:44.118741312 +0000 UTC"
Oct 31 07:28:33.711310 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.711252    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15561h31m10.407493383s"
Oct 31 07:28:33.770775 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.770688    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w" event={"ID":"a329ba42-e8f3-43ed-9355-d906a577e176","Type":"ContainerStarted","Data":"064d02052d83429e3515ed4ca913ccbd9fa7652b571971250a6bd9b68cb83930"}
Oct 31 07:28:33.773038 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.773009    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fz4tp" event={"ID":"4acf6e61-938d-495e-92ea-c1bc343b9406","Type":"ContainerStarted","Data":"4ae2672ff707f35394a0a8bfb496ed32c358570deaf62300c91b68bcbaba8863"}
Oct 31 07:28:33.777469 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.777418    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-49r2r" event={"ID":"5b3d13df-dc50-41c8-a7cd-051fb9934a34","Type":"ContainerStarted","Data":"2f9822d2e207c346eccf487923220859228defbdb9a2878a100a2071ca3c7e18"}
Oct 31 07:28:33.785303 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.785272    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerStarted","Data":"2e27b0d544e40ec8f0656b7c9c8c64a5d7178ffcd9dd6cbc0528e661514171bc"}
Oct 31 07:28:33.788866 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.788837    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbln7" event={"ID":"6121aa17-a37d-47de-af76-08c2aa174568","Type":"ContainerStarted","Data":"62fb044edc2c1ec66daa935b2fadcf4f37117b386ba45dc02767f6c5fa474cd1"}
Oct 31 07:28:33.796084 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.796054    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hmj2s" event={"ID":"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca","Type":"ContainerStarted","Data":"e3e2b63ff27a4d5fab22a0ae75d10d10781aa0169521f6f236ce0748ef466767"}
Oct 31 07:28:33.810781 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.810741    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6859w" event={"ID":"3432e0f3-ae43-4815-a8b1-38e7b7c0222d","Type":"ContainerStarted","Data":"0a219d65cff1ea71f8e7078396466c39bf362b51439e959c770cc7e98adf00ec"}
Oct 31 07:28:33.815040 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.815012    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"a42df5a3903764c9a9afa19fe2c78097","Type":"ContainerStarted","Data":"faec1956febef47147e3b5468f6d9ae9f0d2c6fd64c8dd134c11a8a456b4e669"}
Oct 31 07:28:33.833716 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.833683    2572 generic.go:356] "Generic (PLEG): container finished" podID="d51efa451cfc3022379daa87dd3ede24" containerID="7c422842b213567f5f2c98eb152b1b8c978b3d493d6ba84ae85baa90c9497071" exitCode=0
Oct 31 07:28:33.834000 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.833932    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"d51efa451cfc3022379daa87dd3ede24","Type":"ContainerDied","Data":"7c422842b213567f5f2c98eb152b1b8c978b3d493d6ba84ae85baa90c9497071"}
Oct 31 07:28:33.841030 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.841004    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"e5b849564cdfc431c9a4da01bb4b99d3c869a00c462b30f2177a77cef3636660"}
Oct 31 07:28:33.847621 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.847583    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrtcz" event={"ID":"021b18d4-0686-43c1-9faf-b1ca555d22a3","Type":"ContainerStarted","Data":"827873cf42285fdb51dcb79e7d1a31808677f60b5d944e71ed938f93df9a0859"}
Oct 31 07:28:33.852639 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:33.851555    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal" podStartSLOduration=1.851540517 podStartE2EDuration="1.851540517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:33.831128777 +0000 UTC m=+3.554474133" watchObservedRunningTime="2026-10-31 07:28:33.851540517 +0000 UTC m=+3.574885866"
Oct 31 07:28:34.305623 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.305218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:34.305623 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.305501    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:34.305623 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.305574    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret podName:9420864f-c0f7-47d3-ad36-b4b15ba44b57 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:36.305554584 +0000 UTC m=+6.028899970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret") pod "global-pull-secret-syncer-l4tmf" (UID: "9420864f-c0f7-47d3-ad36-b4b15ba44b57") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:34.406527 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.406463    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:34.406527 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.406517    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:34.406746 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.406651    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:34.406746 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.406697    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:36.406684098 +0000 UTC m=+6.130029435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:34.408887 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.406982    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:34.408887 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.406996    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:34.408887 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.407005    2572 projected.go:196] Error preparing data for projected volume kube-api-access-j988n for pod openshift-network-diagnostics/network-check-target-d5t72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:34.408887 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.407034    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n podName:686827ed-5eb8-4cdc-aeca-176b015eb4f8 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:36.407026484 +0000 UTC m=+6.130371808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j988n" (UniqueName: "kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n") pod "network-check-target-d5t72" (UID: "686827ed-5eb8-4cdc-aeca-176b015eb4f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:34.712121 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.712045    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:32 +0000 UTC" deadline="2028-06-04 00:07:57.397682299 +0000 UTC"
Oct 31 07:28:34.712121 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.712106    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13960h39m22.685581239s"
Oct 31 07:28:34.761268 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.760314    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:34.761268 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.760447    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:34.761268 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.760872    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:34.761268 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.760984    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:34.761268 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.761077    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:34.761268 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:34.761149    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:34.866392 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:34.866315    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"d51efa451cfc3022379daa87dd3ede24","Type":"ContainerStarted","Data":"a4f12a608bedb8144980dc3e98093b064be94fc045f3de01ba518691c9d5949a"}
Oct 31 07:28:35.897987 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:35.897949    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerStarted","Data":"8eef32242a5cc44820dbec330d94f8b1b2fe7ae49a81caa6f1b149601328161e"}
Oct 31 07:28:35.919706 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:35.919646    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal" podStartSLOduration=3.919628098 podStartE2EDuration="3.919628098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:34.885267534 +0000 UTC m=+4.608612895" watchObservedRunningTime="2026-10-31 07:28:35.919628098 +0000 UTC m=+5.642973442"
Oct 31 07:28:36.329579 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:36.329534    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:36.329741 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.329729    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:36.329802 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.329790    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret podName:9420864f-c0f7-47d3-ad36-b4b15ba44b57 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.329773645 +0000 UTC m=+10.053118975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret") pod "global-pull-secret-syncer-l4tmf" (UID: "9420864f-c0f7-47d3-ad36-b4b15ba44b57") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:36.430641 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:36.430601    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:36.430822 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:36.430662    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:36.430822 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.430814    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:36.430925 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.430877    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.430857925 +0000 UTC m=+10.154203276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:36.431454 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.431309    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:36.431454 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.431336    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:36.431454 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.431349    2572 projected.go:196] Error preparing data for projected volume kube-api-access-j988n for pod openshift-network-diagnostics/network-check-target-d5t72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:36.431454 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.431394    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n podName:686827ed-5eb8-4cdc-aeca-176b015eb4f8 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.431378531 +0000 UTC m=+10.154723872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j988n" (UniqueName: "kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n") pod "network-check-target-d5t72" (UID: "686827ed-5eb8-4cdc-aeca-176b015eb4f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:36.762104 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:36.762071    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:36.762290 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.762220    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:36.762619 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:36.762597    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:36.762718 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.762694    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:36.762779 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:36.762750    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:36.762855 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:36.762832    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:36.912459 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:36.912052    2572 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="8eef32242a5cc44820dbec330d94f8b1b2fe7ae49a81caa6f1b149601328161e" exitCode=0
Oct 31 07:28:36.912459 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:36.912113    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerDied","Data":"8eef32242a5cc44820dbec330d94f8b1b2fe7ae49a81caa6f1b149601328161e"}
Oct 31 07:28:37.919755 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:37.919710    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbln7" event={"ID":"6121aa17-a37d-47de-af76-08c2aa174568","Type":"ContainerStarted","Data":"1283480de976bec895c3e59683d542d4a00d5f65e3e7402ae4401d5985ceb293"}
Oct 31 07:28:38.761386 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:38.761359    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:38.761590 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:38.761508    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:38.761590 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:38.761359    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:38.761725 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:38.761628    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:38.761725 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:38.761653    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:38.761830 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:38.761755    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:38.924507 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:38.924401    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w" event={"ID":"a329ba42-e8f3-43ed-9355-d906a577e176","Type":"ContainerStarted","Data":"94f4f99f6b2599f6260572321accabf4256f3e3ea36bccd9d3ec4ceb91f8b1c5"}
Oct 31 07:28:38.928487 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:38.928454    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fz4tp" event={"ID":"4acf6e61-938d-495e-92ea-c1bc343b9406","Type":"ContainerStarted","Data":"3ee5e5639af3a4e2103b9455a8a19f8a400707169878d935d811837c6fd0a618"}
Oct 31 07:28:38.943470 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:38.943160    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sbln7" podStartSLOduration=5.266651968 podStartE2EDuration="8.943143041s" totalImagesPullingTime="3.676491073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:37.936049486 +0000 UTC m=+7.659394831" watchObservedRunningTime="2026-10-31 07:28:38.943143041 +0000 UTC m=+8.666488389"
Oct 31 07:28:40.363195 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.363144    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:40.363660 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.363304    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:40.363660 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.363353    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret podName:9420864f-c0f7-47d3-ad36-b4b15ba44b57 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:48.363339289 +0000 UTC m=+18.086684627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret") pod "global-pull-secret-syncer-l4tmf" (UID: "9420864f-c0f7-47d3-ad36-b4b15ba44b57") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:40.463761 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.463721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:40.463928 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.463829    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:40.463992 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.463972    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:40.464113 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.463998    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:40.464113 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.464012    2572 projected.go:196] Error preparing data for projected volume kube-api-access-j988n for pod openshift-network-diagnostics/network-check-target-d5t72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:40.464113 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.464071    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n podName:686827ed-5eb8-4cdc-aeca-176b015eb4f8 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:48.464052775 +0000 UTC m=+18.187398123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j988n" (UniqueName: "kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n") pod "network-check-target-d5t72" (UID: "686827ed-5eb8-4cdc-aeca-176b015eb4f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:40.464535 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.464516    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:40.464596 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.464572    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:48.464557099 +0000 UTC m=+18.187902442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:40.551808 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.551773    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 07:28:40.762323 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.762295    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:40.762413 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.762319    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:40.762413 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.762254    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:28:40.551803187Z","UUID":"7cddb9d9-f987-4513-869f-606a46854bf2","Handler":null,"Name":"","Endpoint":""}
Oct 31 07:28:40.762522 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.762420    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:40.762522 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.762478    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:40.762616 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.762562    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:40.762661 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:40.762628    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:40.764496 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.764475    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 07:28:40.764610 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.764506    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 07:28:40.936600 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.936562    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hmj2s" event={"ID":"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca","Type":"ContainerStarted","Data":"ab5c55b4758860851753e75fba449d71efa8366c27f5c7780540af5dfe9a11bd"}
Oct 31 07:28:40.944358 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.944291    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w" event={"ID":"a329ba42-e8f3-43ed-9355-d906a577e176","Type":"ContainerStarted","Data":"c26716ad8f027c6059c1db3725d4f58607b673ea56b04e882467e317c12d2bf0"}
Oct 31 07:28:40.953466 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.953397    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hmj2s" podStartSLOduration=3.714233117 podStartE2EDuration="10.953379725s" totalImagesPullingTime="7.239146608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:40.953343754 +0000 UTC m=+10.676689100" watchObservedRunningTime="2026-10-31 07:28:40.953379725 +0000 UTC m=+10.676725072"
Oct 31 07:28:40.954321 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.954286    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fz4tp" podStartSLOduration=5.087365356 podStartE2EDuration="9.954272931s" totalImagesPullingTime="4.866907575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:38.943885861 +0000 UTC m=+8.667231206" watchObservedRunningTime="2026-10-31 07:28:40.954272931 +0000 UTC m=+10.677618278"
Oct 31 07:28:40.968532 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.968511    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:40.969449 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:40.969430    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:41.949898 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:41.949649    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w" event={"ID":"a329ba42-e8f3-43ed-9355-d906a577e176","Type":"ContainerStarted","Data":"c08efe1708f32e9c08fffdbab3fa3be6dc65f7f730588993d66dd47c35208f3a"}
Oct 31 07:28:41.952035 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:41.952010    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-49r2r" event={"ID":"5b3d13df-dc50-41c8-a7cd-051fb9934a34","Type":"ContainerStarted","Data":"0c765802d4aa72266a78745ec019af3464a69d4193085b07963a7d162bd4b3b2"}
Oct 31 07:28:41.953925 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:41.953900    2572 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="d584ae5caedcbfdfa72a819e9e2c2aa172e048053d37416a2a02dd078b684d93" exitCode=0
Oct 31 07:28:41.954042 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:41.953978    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerDied","Data":"d584ae5caedcbfdfa72a819e9e2c2aa172e048053d37416a2a02dd078b684d93"}
Oct 31 07:28:41.967485 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:41.967438    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w" podStartSLOduration=2.958049495 podStartE2EDuration="10.96742614s" totalImagesPullingTime="8.009376645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:41.967281266 +0000 UTC m=+11.690626612" watchObservedRunningTime="2026-10-31 07:28:41.96742614 +0000 UTC m=+11.690771486"
Oct 31 07:28:41.981761 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:41.981719    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-49r2r" podStartSLOduration=3.181929517 podStartE2EDuration="10.981707595s" totalImagesPullingTime="7.799778078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:41.981506713 +0000 UTC m=+11.704852060" watchObservedRunningTime="2026-10-31 07:28:41.981707595 +0000 UTC m=+11.705052940"
Oct 31 07:28:42.759173 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:42.759145    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:42.759373 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:42.759345    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:42.759689 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:42.759670    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:42.759801 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:42.759777    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:42.759925 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:42.759841    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:42.759925 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:42.759916    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:42.958556 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:42.958520    2572 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="3224c9c218b40edab37a55b38b4068b9733e50e91b71655947b7d327ba20c78a" exitCode=0
Oct 31 07:28:42.958972 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:42.958605    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerDied","Data":"3224c9c218b40edab37a55b38b4068b9733e50e91b71655947b7d327ba20c78a"}
Oct 31 07:28:42.960497 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:42.960471    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6859w" event={"ID":"3432e0f3-ae43-4815-a8b1-38e7b7c0222d","Type":"ContainerStarted","Data":"c809189c095fddd97e8bb7feeeae08c18ff3ef8c5360e2ede1ee96f603896560"}
Oct 31 07:28:42.991858 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:42.991806    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6859w" podStartSLOduration=5.227771356 podStartE2EDuration="12.991789454s" totalImagesPullingTime="7.764018098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:42.991746414 +0000 UTC m=+12.715091849" watchObservedRunningTime="2026-10-31 07:28:42.991789454 +0000 UTC m=+12.715134803"
Oct 31 07:28:43.964374 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:43.964338    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrtcz" event={"ID":"021b18d4-0686-43c1-9faf-b1ca555d22a3","Type":"ContainerStarted","Data":"c07db72e0e3f40b128c0fc9165062846fcd426127d9069109105a7c715d99fa9"}
Oct 31 07:28:43.981910 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:43.981865    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nrtcz" podStartSLOduration=2.960814317 podStartE2EDuration="12.981849125s" totalImagesPullingTime="10.021034808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:43.981378041 +0000 UTC m=+13.704723388" watchObservedRunningTime="2026-10-31 07:28:43.981849125 +0000 UTC m=+13.705194480"
Oct 31 07:28:44.762668 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:44.762635    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:44.762831 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:44.762712    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:44.762831 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:44.762811    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:44.762925 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:44.762867    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:44.763022 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:44.762969    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:44.763080 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:44.763063    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:44.969443 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:44.969400    2572 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="97df3bdf7b15687a90d7b9690cebe48b5672c8bc66f5c3778f57e5a69d876968" exitCode=0
Oct 31 07:28:44.970031 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:44.969443    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerDied","Data":"97df3bdf7b15687a90d7b9690cebe48b5672c8bc66f5c3778f57e5a69d876968"}
Oct 31 07:28:46.759913 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:46.759884    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:46.759913 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:46.759899    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:46.760468 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:46.759893    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:46.760468 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:46.759996    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:46.760468 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:46.760110    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:46.760468 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:46.760198    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:46.976254 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:46.976074    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerStarted","Data":"4462d7dc2a7fc5ac76a8c56f6fe9f37d8e434e1fade634c7b84cc3d8b76fac07"}
Oct 31 07:28:47.980753 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:47.980576    2572 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="4462d7dc2a7fc5ac76a8c56f6fe9f37d8e434e1fade634c7b84cc3d8b76fac07" exitCode=0
Oct 31 07:28:47.981222 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:47.980658    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerDied","Data":"4462d7dc2a7fc5ac76a8c56f6fe9f37d8e434e1fade634c7b84cc3d8b76fac07"}
Oct 31 07:28:47.983825 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:47.983806    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"b74a369fdc0efc5569f3e1be96e0e589113f18c08ea400131018975830b768c9"}
Oct 31 07:28:47.983905 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:47.983870    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"53bddbb7ca79b45e1adca61688bf693b84c5ce7ae80ce4d4892207f41229f6c9"}
Oct 31 07:28:47.983940 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:47.983914    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"1bd207c2f226bee743d80b0589e4514abf21bb63d1317b5abcee5d9c142a983f"}
Oct 31 07:28:47.983972 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:47.983954    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"5a62995d1c26d16b26b2c74508777476bc0c5ea4b7066a5bf0bbf04312a64c6f"}
Oct 31 07:28:47.984003 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:47.983982    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"cdd31047279402282e06c6ea6116073bbd00fe51defb15d91137d07286c2bc3a"}
Oct 31 07:28:47.984037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:47.984008    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"958cdf9556c3aeea5b8470e75babcd4064e5125e0bf51b6b29a998c273539b54"}
Oct 31 07:28:48.423592 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:48.423562    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:48.423733 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.423695    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:48.423772 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.423750    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret podName:9420864f-c0f7-47d3-ad36-b4b15ba44b57 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:04.42373667 +0000 UTC m=+34.147081995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret") pod "global-pull-secret-syncer-l4tmf" (UID: "9420864f-c0f7-47d3-ad36-b4b15ba44b57") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:48.524858 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:48.524825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:48.524858 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:48.524865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:48.525048 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.524953    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:48.525048 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.524974    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:48.525048 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.524995    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:48.525048 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.525002    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:04.524989755 +0000 UTC m=+34.248335080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:48.525048 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.525005    2572 projected.go:196] Error preparing data for projected volume kube-api-access-j988n for pod openshift-network-diagnostics/network-check-target-d5t72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:48.525048 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.525045    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n podName:686827ed-5eb8-4cdc-aeca-176b015eb4f8 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:04.525034363 +0000 UTC m=+34.248379687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j988n" (UniqueName: "kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n") pod "network-check-target-d5t72" (UID: "686827ed-5eb8-4cdc-aeca-176b015eb4f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:48.762154 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:48.762074    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:48.762316 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:48.762159    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:48.762316 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:48.762173    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:48.762316 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.762264    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:48.762417 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.762380    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:48.762460 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:48.762435    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:48.988523 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:48.988491    2572 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="43739b739f16c3c8f3ee47c48c40a13b227f0bdb9a1a63c2f09a364ce7a8354a" exitCode=0
Oct 31 07:28:48.988966 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:48.988546    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerDied","Data":"43739b739f16c3c8f3ee47c48c40a13b227f0bdb9a1a63c2f09a364ce7a8354a"}
Oct 31 07:28:49.994546 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:49.994458    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"2f10b9e4b68a17c7b2bbf4605bfb88342daf002c2bf012ca9e6f76983d029220"}
Oct 31 07:28:49.997388 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:49.997359    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4l58" event={"ID":"0546b914-d103-41a5-9e5b-13c6e41bcd85","Type":"ContainerStarted","Data":"581c1d7d76909ce758bde4ccb24f0b751c02900e8b9cbbe8804809925faa3412"}
Oct 31 07:28:50.020620 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:50.020583    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d4l58" podStartSLOduration=8.700838558 podStartE2EDuration="19.020570051s" totalImagesPullingTime="10.319731493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:50.018971742 +0000 UTC m=+19.742317088" watchObservedRunningTime="2026-10-31 07:28:50.020570051 +0000 UTC m=+19.743915379"
Oct 31 07:28:50.759049 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:50.759019    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:50.759277 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:50.759122    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:50.759277 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:50.759141    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:50.760256 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:50.760238    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:50.760359 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:50.760335    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:50.760412 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:50.760391    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:52.004828 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.004621    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" event={"ID":"735d44d3-7a63-4a36-8278-452074cec97c","Type":"ContainerStarted","Data":"1b5e9a803ec6581d208f1e0bf23e380e514d654836ba9160c62e6182ff495b83"}
Oct 31 07:28:52.005383 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.004923    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:52.005383 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.004964    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:52.020603 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.020577    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:52.020744 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.020698    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:28:52.032941 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.032902    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff" podStartSLOduration=6.8131987370000004 podStartE2EDuration="21.032891964s" totalImagesPullingTime="14.219693227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:52.032558672 +0000 UTC m=+21.755904018" watchObservedRunningTime="2026-10-31 07:28:52.032891964 +0000 UTC m=+21.756237351"
Oct 31 07:28:52.759998 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.759970    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:52.760159 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.759982    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:52.760159 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:52.760064    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:52.760159 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:52.760143    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:52.760280 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:52.760202    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:52.760280 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:52.760262    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:53.007410 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:53.007381    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:53.558938 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:53.558907    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l4tmf"]
Oct 31 07:28:53.559134 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:53.559015    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:53.559134 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:53.559122    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:53.561815 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:53.561790    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svhj9"]
Oct 31 07:28:53.561915 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:53.561881    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:53.562005 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:53.561984    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:53.564502 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:53.564483    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d5t72"]
Oct 31 07:28:53.564589 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:53.564555    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:53.564635 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:53.564621    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:54.008927 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:54.008901    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:55.759202 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:55.759147    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:55.759202 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:55.759201    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:55.759722 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:55.759263    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:55.759722 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:55.759312    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:55.759722 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:55.759377    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:55.759722 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:55.759439    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:57.157137 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:57.157105    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:57.157566 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:57.157247    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:57.157962 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:57.157908    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:28:57.760125 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:57.760089    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:57.760125 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:57.760126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:57.760382 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:57.760134    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:57.760382 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:57.760224    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:57.760382 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:57.760353    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:28:57.760535 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:57.760449    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:59.759856 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:59.759825    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:28:59.760264 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:59.759902    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:28:59.760264 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:28:59.759928    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:28:59.760264 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:59.760009    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svhj9" podUID="b5bca13e-12d2-4673-8223-92f7755eac90"
Oct 31 07:28:59.760264 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:59.760072    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l4tmf" podUID="9420864f-c0f7-47d3-ad36-b4b15ba44b57"
Oct 31 07:28:59.760264 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:28:59.760143    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5t72" podUID="686827ed-5eb8-4cdc-aeca-176b015eb4f8"
Oct 31 07:29:00.063000 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.062970    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeReady"
Oct 31 07:29:00.063136 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.063088    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 07:29:00.127531 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.127451    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-22c6v"]
Oct 31 07:29:00.136762 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.136736    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5nzmp"]
Oct 31 07:29:00.136923 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.136839    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-22c6v"
Oct 31 07:29:00.140763 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.140576    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ch85h"]
Oct 31 07:29:00.140863 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.140683    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5nzmp"
Oct 31 07:29:00.143109 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.143090    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.144059 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.144039    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.144806 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.144791    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rjjcf"]
Oct 31 07:29:00.144948 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.144932    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ch85h"
Oct 31 07:29:00.147229 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.147212    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f6f75f655-wgvm2"]
Oct 31 07:29:00.147348 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.147313    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.151216 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.149789    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"]
Oct 31 07:29:00.151216 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.150550    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.155069 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.155047    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"]
Oct 31 07:29:00.156846 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.156826    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.160256 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.160235    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"]
Oct 31 07:29:00.161167 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.161146    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.169059 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.168148    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 07:29:00.169439 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.169400    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-48flb\""
Oct 31 07:29:00.171342 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.171321    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jgczn"]
Oct 31 07:29:00.171495 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.171478    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.186355 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.186242    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"]
Oct 31 07:29:00.188925 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.188899    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:00.192967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.192944    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b69fcb956-87gvh"]
Oct 31 07:29:00.193098 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.193021    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.196596 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.196578    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"]
Oct 31 07:29:00.196742 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.196724    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.200243 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.200222    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"]
Oct 31 07:29:00.200353 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.200335    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"
Oct 31 07:29:00.203518 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.203503    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f4d8d5849-jrvx7"]
Oct 31 07:29:00.203624 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.203608    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.206285 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.206266    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"]
Oct 31 07:29:00.206368 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.206355    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.208277 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.208260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtlv\" (UniqueName: \"kubernetes.io/projected/385e4d20-8bd7-4933-8a7d-4359e0301d89-kube-api-access-tmtlv\") pod \"kas-connection-checker-685895c7c9-22c6v\" (UID: \"385e4d20-8bd7-4933-8a7d-4359e0301d89\") " pod="kube-system/kas-connection-checker-685895c7c9-22c6v"
Oct 31 07:29:00.209448 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.209431    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5nzmp"]
Oct 31 07:29:00.209530 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.209466    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ch85h"]
Oct 31 07:29:00.209669 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.209539    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"]
Oct 31 07:29:00.209669 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.209568    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-46cpb"]
Oct 31 07:29:00.210280 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.209831    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"
Oct 31 07:29:00.214007 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.213989    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-22c6v"]
Oct 31 07:29:00.214067 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.214027    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"]
Oct 31 07:29:00.214067 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.214051    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w9f8b"]
Oct 31 07:29:00.214132 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.214079    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.228768 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228749    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"]
Oct 31 07:29:00.228858 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228779    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6f75f655-wgvm2"]
Oct 31 07:29:00.228858 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228798    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"]
Oct 31 07:29:00.228967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228857    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:00.228967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228902    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"]
Oct 31 07:29:00.228967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228921    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jgczn"]
Oct 31 07:29:00.228967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228933    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-46cpb"]
Oct 31 07:29:00.228967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228952    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"]
Oct 31 07:29:00.228967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228968    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rjjcf"]
Oct 31 07:29:00.229258 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228982    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w9f8b"]
Oct 31 07:29:00.229258 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.228995    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b69fcb956-87gvh"]
Oct 31 07:29:00.229258 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.229023    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4d8d5849-jrvx7"]
Oct 31 07:29:00.229258 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.229050    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"]
Oct 31 07:29:00.244077 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244052    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.244167 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244150    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 07:29:00.244167 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244052    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 07:29:00.244292 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244161    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.244292 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244214    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.244292 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244246    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.244292 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244256    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.244527 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244355    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.244527 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244417    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.244527 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244447    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.244527 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244478    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.244691 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244660    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.244766 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244744    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.244885 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244809    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.244885 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244853    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.245026 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244889    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 07:29:00.245026 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244918    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 07:29:00.245026 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.244944    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 07:29:00.245611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.245592    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.245611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.245607    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 07:29:00.245611 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.245598    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.245841 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.245622    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.246233 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.246217    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.246336 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.246222    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 07:29:00.246336 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.246261    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 07:29:00.246336 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.246270    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.247101 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.247086    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 07:29:00.247911 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.247871    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 07:29:00.253094 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.253078    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 07:29:00.263037 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.263010    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 07:29:00.263174 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.263149    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 07:29:00.263255 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.263243    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 07:29:00.263309 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.263278    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 07:29:00.263366 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.263321    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 07:29:00.263945 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.263926    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 07:29:00.264219 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264178    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ps9gq\""
Oct 31 07:29:00.264302 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264196    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hfwqd\""
Oct 31 07:29:00.264302 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264244    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lj5js\""
Oct 31 07:29:00.264302 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264272    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 07:29:00.264302 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264292    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 07:29:00.264476 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264336    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9wzll\""
Oct 31 07:29:00.264476 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264348    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 07:29:00.264476 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264449    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 07:29:00.264614 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264504    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 07:29:00.264614 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264593    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p7ztr\""
Oct 31 07:29:00.264779 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264763    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 07:29:00.264869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264785    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5tc29\""
Oct 31 07:29:00.264869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264798    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 07:29:00.264869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264822    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 07:29:00.264869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264826    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6ctpv\""
Oct 31 07:29:00.264869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264834    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p4qcx\""
Oct 31 07:29:00.264869 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264871    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 07:29:00.265121 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264875    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gd7dx\""
Oct 31 07:29:00.265121 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264921    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zzbc9\""
Oct 31 07:29:00.265121 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.264953    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-zwdw4\""
Oct 31 07:29:00.265121 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.265028    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jwmk9\""
Oct 31 07:29:00.265121 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.265087    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 07:29:00.265382 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.265227    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 07:29:00.308626 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.308772 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308632    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgq9\" (UniqueName: \"kubernetes.io/projected/8b210005-d749-4ec8-ad41-bc7afaeb7011-kube-api-access-4wgq9\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.308772 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308661    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-stats-auth\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.308772 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308682    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7bd\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.308772 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b210005-d749-4ec8-ad41-bc7afaeb7011-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.308772 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308748    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtlv\" (UniqueName: \"kubernetes.io/projected/385e4d20-8bd7-4933-8a7d-4359e0301d89-kube-api-access-tmtlv\") pod \"kas-connection-checker-685895c7c9-22c6v\" (UID: \"385e4d20-8bd7-4933-8a7d-4359e0301d89\") " pod="kube-system/kas-connection-checker-685895c7c9-22c6v"
Oct 31 07:29:00.309016 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308770    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63139743-194a-4517-9eba-62a858a3b649-cert\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:00.309016 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308809    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqf8w\" (UniqueName: \"kubernetes.io/projected/1820478c-a566-4c78-b063-e8ceb9c6e221-kube-api-access-cqf8w\") pod \"kas-connection-checker-685895c7c9-5nzmp\" (UID: \"1820478c-a566-4c78-b063-e8ceb9c6e221\") " pod="kube-system/kas-connection-checker-685895c7c9-5nzmp"
Oct 31 07:29:00.309016 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308835    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brvk\" (UniqueName: \"kubernetes.io/projected/3adfe0f6-05ab-44f9-8884-06c021c42f5a-kube-api-access-2brvk\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.309016 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308866    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.309016 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308894    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmn5\" (UniqueName: \"kubernetes.io/projected/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-kube-api-access-vwmn5\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.309016 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.309016 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.308950    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.309016 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309005    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1a70f5-55db-4e36-959d-95bae0872fea-serving-cert\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309167    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309217    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-tmp\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309240    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309290    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309323    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774lj\" (UniqueName: \"kubernetes.io/projected/9a1a70f5-55db-4e36-959d-95bae0872fea-kube-api-access-774lj\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.309379 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309374    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.309667 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7dpq\" (UniqueName: \"kubernetes.io/projected/c64dab3d-c1ed-4828-b71f-d80b1f7fab00-kube-api-access-m7dpq\") pod \"network-check-source-6f78d59f99-9sbwg\" (UID: \"c64dab3d-c1ed-4828-b71f-d80b1f7fab00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"
Oct 31 07:29:00.309667 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e684ad1d-ea28-4d75-aca0-8c12227ffada-serving-cert\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.309667 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309601    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.309667 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309632    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.309667 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309657    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4fk2\" (UniqueName: \"kubernetes.io/projected/59cf0458-de23-4a5e-884f-4a583e78ff38-kube-api-access-n4fk2\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.309836 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309754    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8d86\" (UniqueName: \"kubernetes.io/projected/63139743-194a-4517-9eba-62a858a3b649-kube-api-access-d8d86\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:00.309836 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309782    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59cf0458-de23-4a5e-884f-4a583e78ff38-config\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.309936 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.309936 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg4vv\" (UniqueName: \"kubernetes.io/projected/83e1b5fd-c06c-45a2-9c74-f29515e920ff-kube-api-access-cg4vv\") pod \"volume-data-source-validator-58c9f5f46d-t6dgh\" (UID: \"83e1b5fd-c06c-45a2-9c74-f29515e920ff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"
Oct 31 07:29:00.310032 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309937    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.310032 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.309977    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.310032 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310004    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:00.310224 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310029    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdmk\" (UniqueName: \"kubernetes.io/projected/bd4df54c-65e4-4dba-943c-17219493fc36-kube-api-access-lcdmk\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.310224 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310070    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.310224 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c82b956f-d109-4a4e-9a90-f298f5dcaf11-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:00.310224 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310125    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3adfe0f6-05ab-44f9-8884-06c021c42f5a-tmp-dir\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.310224 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310167    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-snapshots\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.310224 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlnk\" (UniqueName: \"kubernetes.io/projected/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-kube-api-access-hjlnk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.310467 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310264    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pdw\" (UniqueName: \"kubernetes.io/projected/0d264640-a684-42bd-8787-86d2382a7eb4-kube-api-access-v5pdw\") pod \"kas-connection-checker-685895c7c9-ch85h\" (UID: \"0d264640-a684-42bd-8787-86d2382a7eb4\") " pod="kube-system/kas-connection-checker-685895c7c9-ch85h"
Oct 31 07:29:00.310467 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310312    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.310467 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310335    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-trusted-ca\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.310467 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9xm\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.310467 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310432    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.310467 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310461    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.310712 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310509    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-service-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.310712 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310555    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59cf0458-de23-4a5e-884f-4a583e78ff38-serving-cert\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.310712 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-default-certificate\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.310712 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310673    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.310712 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310690    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.310712 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310707    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3adfe0f6-05ab-44f9-8884-06c021c42f5a-config-volume\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.310955 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310722    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-config\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.310955 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310773    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hgs9\" (UniqueName: \"kubernetes.io/projected/e684ad1d-ea28-4d75-aca0-8c12227ffada-kube-api-access-5hgs9\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.310955 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.310801    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.321795 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.321768    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtlv\" (UniqueName: \"kubernetes.io/projected/385e4d20-8bd7-4933-8a7d-4359e0301d89-kube-api-access-tmtlv\") pod \"kas-connection-checker-685895c7c9-22c6v\" (UID: \"385e4d20-8bd7-4933-8a7d-4359e0301d89\") " pod="kube-system/kas-connection-checker-685895c7c9-22c6v"
Oct 31 07:29:00.411390 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411309    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlnk\" (UniqueName: \"kubernetes.io/projected/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-kube-api-access-hjlnk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.411390 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411356    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pdw\" (UniqueName: \"kubernetes.io/projected/0d264640-a684-42bd-8787-86d2382a7eb4-kube-api-access-v5pdw\") pod \"kas-connection-checker-685895c7c9-ch85h\" (UID: \"0d264640-a684-42bd-8787-86d2382a7eb4\") " pod="kube-system/kas-connection-checker-685895c7c9-ch85h"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411437    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-trusted-ca\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411458    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9xm\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411475    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411492    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411520    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-service-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411544    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59cf0458-de23-4a5e-884f-4a583e78ff38-serving-cert\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411569    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-default-certificate\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.411615 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411603    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.412023 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.411627    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.412080 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412038    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.412268 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412245    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3adfe0f6-05ab-44f9-8884-06c021c42f5a-config-volume\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.412366 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-config\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.412366 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412289    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.412366 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412317    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgs9\" (UniqueName: \"kubernetes.io/projected/e684ad1d-ea28-4d75-aca0-8c12227ffada-kube-api-access-5hgs9\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.412366 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412344    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.412562 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412387    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.412562 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412453    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-trusted-ca\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.412562 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.412517    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:00.91249412 +0000 UTC m=+30.635839463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:00.412562 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412549    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgq9\" (UniqueName: \"kubernetes.io/projected/8b210005-d749-4ec8-ad41-bc7afaeb7011-kube-api-access-4wgq9\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.412750 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-stats-auth\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.412750 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412661    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-service-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.412750 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412728    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.412885 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.412847    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:00.412885 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.412883    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls podName:8b210005-d749-4ec8-ad41-bc7afaeb7011 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:00.912872388 +0000 UTC m=+30.636217715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qrsxv" (UID: "8b210005-d749-4ec8-ad41-bc7afaeb7011") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:00.412970 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7bd\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.412970 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412936    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3adfe0f6-05ab-44f9-8884-06c021c42f5a-config-volume\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.413070 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.412941    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b210005-d749-4ec8-ad41-bc7afaeb7011-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.413070 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413031    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63139743-194a-4517-9eba-62a858a3b649-cert\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:00.413164 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413066    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqf8w\" (UniqueName: \"kubernetes.io/projected/1820478c-a566-4c78-b063-e8ceb9c6e221-kube-api-access-cqf8w\") pod \"kas-connection-checker-685895c7c9-5nzmp\" (UID: \"1820478c-a566-4c78-b063-e8ceb9c6e221\") " pod="kube-system/kas-connection-checker-685895c7c9-5nzmp"
Oct 31 07:29:00.413164 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brvk\" (UniqueName: \"kubernetes.io/projected/3adfe0f6-05ab-44f9-8884-06c021c42f5a-kube-api-access-2brvk\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.413164 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413134    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-config\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.413164 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.413402 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413220    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmn5\" (UniqueName: \"kubernetes.io/projected/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-kube-api-access-vwmn5\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.413402 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.413402 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413278    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.413402 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413306    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1a70f5-55db-4e36-959d-95bae0872fea-serving-cert\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.413402 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.413650 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.413424    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:00.413650 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.413475    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:00.913461781 +0000 UTC m=+30.636807120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : secret "router-metrics-certs-default" not found
Oct 31 07:29:00.413650 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.413554    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:00.413650 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.413589    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:00.913577847 +0000 UTC m=+30.636923193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:00.413848 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413647    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b210005-d749-4ec8-ad41-bc7afaeb7011-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.413977 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.413954    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.414038 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414020    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.414090 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414056    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-tmp\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.414143 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.414143 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414137    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.414251 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414164    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.414251 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774lj\" (UniqueName: \"kubernetes.io/projected/9a1a70f5-55db-4e36-959d-95bae0872fea-kube-api-access-774lj\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.414251 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414245    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.414396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414281    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7dpq\" (UniqueName: \"kubernetes.io/projected/c64dab3d-c1ed-4828-b71f-d80b1f7fab00-kube-api-access-m7dpq\") pod \"network-check-source-6f78d59f99-9sbwg\" (UID: \"c64dab3d-c1ed-4828-b71f-d80b1f7fab00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"
Oct 31 07:29:00.414396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e684ad1d-ea28-4d75-aca0-8c12227ffada-serving-cert\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.414396 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414378    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.414526 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.414526 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414461    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fk2\" (UniqueName: \"kubernetes.io/projected/59cf0458-de23-4a5e-884f-4a583e78ff38-kube-api-access-n4fk2\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.414616 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414566    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8d86\" (UniqueName: \"kubernetes.io/projected/63139743-194a-4517-9eba-62a858a3b649-kube-api-access-d8d86\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:00.414616 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414592    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59cf0458-de23-4a5e-884f-4a583e78ff38-config\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.414616 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414606    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-tmp\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.414758 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414625    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.414758 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414652    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg4vv\" (UniqueName: \"kubernetes.io/projected/83e1b5fd-c06c-45a2-9c74-f29515e920ff-kube-api-access-cg4vv\") pod \"volume-data-source-validator-58c9f5f46d-t6dgh\" (UID: \"83e1b5fd-c06c-45a2-9c74-f29515e920ff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"
Oct 31 07:29:00.414758 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.414758 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.414696    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:00.414758 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.414709    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:00.414758 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414711    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.414758 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414739    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:00.414758 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.414749    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:00.914735641 +0000 UTC m=+30.638080981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:00.415110 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.414841    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:00.415110 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.414885    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert podName:c82b956f-d109-4a4e-9a90-f298f5dcaf11 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:00.914871971 +0000 UTC m=+30.638217321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jgczn" (UID: "c82b956f-d109-4a4e-9a90-f298f5dcaf11") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:00.415110 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414906    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdmk\" (UniqueName: \"kubernetes.io/projected/bd4df54c-65e4-4dba-943c-17219493fc36-kube-api-access-lcdmk\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.415110 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414942    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.415110 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.414971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c82b956f-d109-4a4e-9a90-f298f5dcaf11-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:00.415110 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.415006    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3adfe0f6-05ab-44f9-8884-06c021c42f5a-tmp-dir\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.415110 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.415034    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-snapshots\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.415455 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.415136    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.415509 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.415487    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.415813 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.415788    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.416101 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.416081    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:00.416175 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.415795    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59cf0458-de23-4a5e-884f-4a583e78ff38-config\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.416175 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.416106    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:00.416175 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.416175    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:00.916157276 +0000 UTC m=+30.639502617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:00.416384 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.416254    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-snapshots\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.416384 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.416299    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.416384 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.416380    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:00.416528 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.416432    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:00.916418501 +0000 UTC m=+30.639763828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:00.416730 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.416709    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.416791 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.416742    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-default-certificate\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.417500 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.417072    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.417500 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.417089    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.417500 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.417365    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3adfe0f6-05ab-44f9-8884-06c021c42f5a-tmp-dir\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.417500 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.417381    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63139743-194a-4517-9eba-62a858a3b649-cert\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:00.417750 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.417541    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.418239 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.417988    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c82b956f-d109-4a4e-9a90-f298f5dcaf11-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:00.418239 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.418198    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.418390 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.418253    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.418390 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.418307    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1a70f5-55db-4e36-959d-95bae0872fea-serving-cert\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.418480 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.418445    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59cf0458-de23-4a5e-884f-4a583e78ff38-serving-cert\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.419321 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.419297    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-stats-auth\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.420289 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.420008    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e684ad1d-ea28-4d75-aca0-8c12227ffada-serving-cert\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.420368 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.420353    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9xm\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.420789 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.420537    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.422692 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.422648    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqf8w\" (UniqueName: \"kubernetes.io/projected/1820478c-a566-4c78-b063-e8ceb9c6e221-kube-api-access-cqf8w\") pod \"kas-connection-checker-685895c7c9-5nzmp\" (UID: \"1820478c-a566-4c78-b063-e8ceb9c6e221\") " pod="kube-system/kas-connection-checker-685895c7c9-5nzmp"
Oct 31 07:29:00.422795 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.422775    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgs9\" (UniqueName: \"kubernetes.io/projected/e684ad1d-ea28-4d75-aca0-8c12227ffada-kube-api-access-5hgs9\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.425955 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.425924    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.427678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.426231    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmn5\" (UniqueName: \"kubernetes.io/projected/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-kube-api-access-vwmn5\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.427678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.425967    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7bd\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.427678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.426619    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgq9\" (UniqueName: \"kubernetes.io/projected/8b210005-d749-4ec8-ad41-bc7afaeb7011-kube-api-access-4wgq9\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.427678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.426730    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.427678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.426901    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlnk\" (UniqueName: \"kubernetes.io/projected/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-kube-api-access-hjlnk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.427678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.426930    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brvk\" (UniqueName: \"kubernetes.io/projected/3adfe0f6-05ab-44f9-8884-06c021c42f5a-kube-api-access-2brvk\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.427678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.427443    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774lj\" (UniqueName: \"kubernetes.io/projected/9a1a70f5-55db-4e36-959d-95bae0872fea-kube-api-access-774lj\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.427678 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.427460    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pdw\" (UniqueName: \"kubernetes.io/projected/0d264640-a684-42bd-8787-86d2382a7eb4-kube-api-access-v5pdw\") pod \"kas-connection-checker-685895c7c9-ch85h\" (UID: \"0d264640-a684-42bd-8787-86d2382a7eb4\") " pod="kube-system/kas-connection-checker-685895c7c9-ch85h"
Oct 31 07:29:00.430949 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.430921    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg4vv\" (UniqueName: \"kubernetes.io/projected/83e1b5fd-c06c-45a2-9c74-f29515e920ff-kube-api-access-cg4vv\") pod \"volume-data-source-validator-58c9f5f46d-t6dgh\" (UID: \"83e1b5fd-c06c-45a2-9c74-f29515e920ff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"
Oct 31 07:29:00.431168 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.431129    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fk2\" (UniqueName: \"kubernetes.io/projected/59cf0458-de23-4a5e-884f-4a583e78ff38-kube-api-access-n4fk2\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.434781 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.434759    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8d86\" (UniqueName: \"kubernetes.io/projected/63139743-194a-4517-9eba-62a858a3b649-kube-api-access-d8d86\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:00.434862 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.434824    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7dpq\" (UniqueName: \"kubernetes.io/projected/c64dab3d-c1ed-4828-b71f-d80b1f7fab00-kube-api-access-m7dpq\") pod \"network-check-source-6f78d59f99-9sbwg\" (UID: \"c64dab3d-c1ed-4828-b71f-d80b1f7fab00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"
Oct 31 07:29:00.435210 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.435166    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdmk\" (UniqueName: \"kubernetes.io/projected/bd4df54c-65e4-4dba-943c-17219493fc36-kube-api-access-lcdmk\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.450895 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.450876    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-22c6v"
Oct 31 07:29:00.463286 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.463261    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5nzmp"
Oct 31 07:29:00.482462 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.482446    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ch85h"
Oct 31 07:29:00.491652 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.491634    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:00.508763 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.508744    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:00.524352 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.524319    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:00.537240 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.537210    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:00.557574 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.557545    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"
Oct 31 07:29:00.575529 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.575481    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"
Oct 31 07:29:00.586909 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.586860    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:00.830124 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.830097    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-22c6v"]
Oct 31 07:29:00.834332 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.834248    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ch85h"]
Oct 31 07:29:00.837430 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.835769    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"]
Oct 31 07:29:00.837430 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.836691    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5nzmp"]
Oct 31 07:29:00.840684 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:00.840650    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59cf0458_de23_4a5e_884f_4a583e78ff38.slice/crio-1e67e7ae3b7a66c55a6ff6eba5d7769dfadfff9b2364ca4735a86ad49a5904ce WatchSource:0}: Error finding container 1e67e7ae3b7a66c55a6ff6eba5d7769dfadfff9b2364ca4735a86ad49a5904ce: Status 404 returned error can't find the container with id 1e67e7ae3b7a66c55a6ff6eba5d7769dfadfff9b2364ca4735a86ad49a5904ce
Oct 31 07:29:00.840873 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:00.840856    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d264640_a684_42bd_8787_86d2382a7eb4.slice/crio-9d0f9e9079fc53a703379391f34ab9b1ebc69d7e349be659863c7e674037a4f3 WatchSource:0}: Error finding container 9d0f9e9079fc53a703379391f34ab9b1ebc69d7e349be659863c7e674037a4f3: Status 404 returned error can't find the container with id 9d0f9e9079fc53a703379391f34ab9b1ebc69d7e349be659863c7e674037a4f3
Oct 31 07:29:00.841969 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:00.841949    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1820478c_a566_4c78_b063_e8ceb9c6e221.slice/crio-bd8b34bfe5082da35ca9028be98db54e34eed727be0429cef3850231413e6886 WatchSource:0}: Error finding container bd8b34bfe5082da35ca9028be98db54e34eed727be0429cef3850231413e6886: Status 404 returned error can't find the container with id bd8b34bfe5082da35ca9028be98db54e34eed727be0429cef3850231413e6886
Oct 31 07:29:00.864555 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.864314    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rjjcf"]
Oct 31 07:29:00.866390 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:00.866363    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode684ad1d_ea28_4d75_aca0_8c12227ffada.slice/crio-703cf2933db32a4c3b02047cff77e6f3dd07ff1341bb2f1702644270240c21c0 WatchSource:0}: Error finding container 703cf2933db32a4c3b02047cff77e6f3dd07ff1341bb2f1702644270240c21c0: Status 404 returned error can't find the container with id 703cf2933db32a4c3b02047cff77e6f3dd07ff1341bb2f1702644270240c21c0
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.922258    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.922306    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.922335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.922376    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.922400    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.922445    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.922464    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.922496    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.922602    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.922612    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.922656    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:01.922644606 +0000 UTC m=+31.645989930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.922967    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923001    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:01.922991714 +0000 UTC m=+31.646337038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923038    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923045    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923066    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:01.923060384 +0000 UTC m=+31.646405707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:00.923463 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923107    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:00.924505 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923127    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert podName:c82b956f-d109-4a4e-9a90-f298f5dcaf11 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:01.923122195 +0000 UTC m=+31.646467519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jgczn" (UID: "c82b956f-d109-4a4e-9a90-f298f5dcaf11") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:00.924505 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923168    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:01.923162561 +0000 UTC m=+31.646507885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:00.924505 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923229    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:00.924505 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923258    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls podName:8b210005-d749-4ec8-ad41-bc7afaeb7011 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:01.923249425 +0000 UTC m=+31.646594755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qrsxv" (UID: "8b210005-d749-4ec8-ad41-bc7afaeb7011") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:00.924505 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923310    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:00.924505 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923341    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:01.92333033 +0000 UTC m=+31.646675654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:00.924505 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923391    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:00.924505 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:00.923418    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:01.923409624 +0000 UTC m=+31.646754951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : secret "router-metrics-certs-default" not found
Oct 31 07:29:00.930932 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.930758    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"]
Oct 31 07:29:00.937461 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:00.937427    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fbf516_95a9_4c2c_a41f_1a053ff98e6b.slice/crio-d03de8ef1759a01a591b57d7b999191414c4e19d05d3db7610a8c67074a59fb9 WatchSource:0}: Error finding container d03de8ef1759a01a591b57d7b999191414c4e19d05d3db7610a8c67074a59fb9: Status 404 returned error can't find the container with id d03de8ef1759a01a591b57d7b999191414c4e19d05d3db7610a8c67074a59fb9
Oct 31 07:29:00.939370 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.939320    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w9f8b"]
Oct 31 07:29:00.944302 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:00.944277    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63139743_194a_4517_9eba_62a858a3b649.slice/crio-e975873444920ca169dec3ea811183ba6de514cd108e34a48d10ac21ae09856f WatchSource:0}: Error finding container e975873444920ca169dec3ea811183ba6de514cd108e34a48d10ac21ae09856f: Status 404 returned error can't find the container with id e975873444920ca169dec3ea811183ba6de514cd108e34a48d10ac21ae09856f
Oct 31 07:29:00.944863 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.944601    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"]
Oct 31 07:29:00.963509 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:00.949025    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1a70f5_55db_4e36_959d_95bae0872fea.slice/crio-498f067da301d7bc89431b26856057279333647defa3ec6ca18e5d5c084384df WatchSource:0}: Error finding container 498f067da301d7bc89431b26856057279333647defa3ec6ca18e5d5c084384df: Status 404 returned error can't find the container with id 498f067da301d7bc89431b26856057279333647defa3ec6ca18e5d5c084384df
Oct 31 07:29:00.979764 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.979735    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"]
Oct 31 07:29:00.996199 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:00.994504    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"]
Oct 31 07:29:01.032839 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.032812    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb" event={"ID":"59cf0458-de23-4a5e-884f-4a583e78ff38","Type":"ContainerStarted","Data":"1e67e7ae3b7a66c55a6ff6eba5d7769dfadfff9b2364ca4735a86ad49a5904ce"}
Oct 31 07:29:01.034284 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.034171    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-22c6v" event={"ID":"385e4d20-8bd7-4933-8a7d-4359e0301d89","Type":"ContainerStarted","Data":"69d2b93f731ff6acb8cdcf0324b0f1a8165b2541fbad187fcd3fb6f855f2cb28"}
Oct 31 07:29:01.034284 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.034225    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-22c6v" event={"ID":"385e4d20-8bd7-4933-8a7d-4359e0301d89","Type":"ContainerStarted","Data":"422e06340d9787fb5b64f74789dea9c9a5ce8a523b043e8f1024a030ab174df2"}
Oct 31 07:29:01.037742 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.037716    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" event={"ID":"9a1a70f5-55db-4e36-959d-95bae0872fea","Type":"ContainerStarted","Data":"498f067da301d7bc89431b26856057279333647defa3ec6ca18e5d5c084384df"}
Oct 31 07:29:01.038816 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.038795    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg" event={"ID":"c64dab3d-c1ed-4828-b71f-d80b1f7fab00","Type":"ContainerStarted","Data":"b66fd7932ec7f3252e9d8c75e32c041e93be091da7d3dd022bb68bddb510b4dd"}
Oct 31 07:29:01.040146 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.040115    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w9f8b" event={"ID":"63139743-194a-4517-9eba-62a858a3b649","Type":"ContainerStarted","Data":"e975873444920ca169dec3ea811183ba6de514cd108e34a48d10ac21ae09856f"}
Oct 31 07:29:01.041475 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.041425    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l" event={"ID":"87fbf516-95a9-4c2c-a41f-1a053ff98e6b","Type":"ContainerStarted","Data":"d03de8ef1759a01a591b57d7b999191414c4e19d05d3db7610a8c67074a59fb9"}
Oct 31 07:29:01.042965 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.042935    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ch85h" event={"ID":"0d264640-a684-42bd-8787-86d2382a7eb4","Type":"ContainerStarted","Data":"67f1b4dddecd982f393017ba9fb3edaa2ed5065ed23437c3817dba6aa0474e80"}
Oct 31 07:29:01.043036 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.042973    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ch85h" event={"ID":"0d264640-a684-42bd-8787-86d2382a7eb4","Type":"ContainerStarted","Data":"9d0f9e9079fc53a703379391f34ab9b1ebc69d7e349be659863c7e674037a4f3"}
Oct 31 07:29:01.044385 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.044361    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5nzmp" event={"ID":"1820478c-a566-4c78-b063-e8ceb9c6e221","Type":"ContainerStarted","Data":"6708b15371ec2a2b9a0e2694ae0357db36cd9d33f2daf4b13cc2c29a25c7dd00"}
Oct 31 07:29:01.044468 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.044405    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5nzmp" event={"ID":"1820478c-a566-4c78-b063-e8ceb9c6e221","Type":"ContainerStarted","Data":"bd8b34bfe5082da35ca9028be98db54e34eed727be0429cef3850231413e6886"}
Oct 31 07:29:01.046683 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.046079    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh" event={"ID":"83e1b5fd-c06c-45a2-9c74-f29515e920ff","Type":"ContainerStarted","Data":"249c027c5adfb01f21ba0fad290fd1b1767eb9b119bec5e0f2653d825f28660b"}
Oct 31 07:29:01.047338 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.047291    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rjjcf" event={"ID":"e684ad1d-ea28-4d75-aca0-8c12227ffada","Type":"ContainerStarted","Data":"703cf2933db32a4c3b02047cff77e6f3dd07ff1341bb2f1702644270240c21c0"}
Oct 31 07:29:01.061603 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.061254    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5nzmp" podStartSLOduration=416.06123962 podStartE2EDuration="6m56.06123962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:01.061043054 +0000 UTC m=+30.784388400" watchObservedRunningTime="2026-10-31 07:29:01.06123962 +0000 UTC m=+30.784584967"
Oct 31 07:29:01.759842 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.759806    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:01.760744 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.760542    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:01.761110 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.761085    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:01.764711 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.764629    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2shdh\""
Oct 31 07:29:01.764711 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.764632    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:29:01.764867 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.764752    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xr95\""
Oct 31 07:29:01.766316 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.766169    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ht7pz\""
Oct 31 07:29:01.766316 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.766295    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:29:01.928233 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.927443    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:01.928233 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.927689    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.933687    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.933754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.933831    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.933862    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.933880    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.933858979 +0000 UTC m=+33.657204317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.933955    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.933963    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.934025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934058    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934099    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls podName:8b210005-d749-4ec8-ad41-bc7afaeb7011 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.934082689 +0000 UTC m=+33.657428028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qrsxv" (UID: "8b210005-d749-4ec8-ad41-bc7afaeb7011") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.934122    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934150    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.934153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934201    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.934173228 +0000 UTC m=+33.657518559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934283    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934379    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:01.934896 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934390    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:01.935778 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934462    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:01.935778 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934500    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.934487456 +0000 UTC m=+33.657832810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : secret "router-metrics-certs-default" not found
Oct 31 07:29:01.935778 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934517    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.934508537 +0000 UTC m=+33.657853864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:01.935778 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934536    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.934526955 +0000 UTC m=+33.657872286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:01.935778 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934552    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert podName:c82b956f-d109-4a4e-9a90-f298f5dcaf11 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.934544167 +0000 UTC m=+33.657889513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jgczn" (UID: "c82b956f-d109-4a4e-9a90-f298f5dcaf11") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:01.935778 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934592    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:01.935778 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934611    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:01.935778 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:01.934653    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.934637519 +0000 UTC m=+33.657982846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:01.981547 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:01.981519    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:02.071571 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:02.071420    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-22c6v" podStartSLOduration=417.071401633 podStartE2EDuration="6m57.071401633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:02.070476511 +0000 UTC m=+31.793821860" watchObservedRunningTime="2026-10-31 07:29:02.071401633 +0000 UTC m=+31.794746981"
Oct 31 07:29:02.086872 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:02.086826    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ch85h" podStartSLOduration=417.086813465 podStartE2EDuration="6m57.086813465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:02.086621981 +0000 UTC m=+31.809967352" watchObservedRunningTime="2026-10-31 07:29:02.086813465 +0000 UTC m=+31.810158811"
Oct 31 07:29:03.957790 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:03.957724    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:03.957790 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:03.957766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:03.957807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:03.957845    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:03.957878    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.957886    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:03.957905    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.957952    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.957934905 +0000 UTC m=+37.681280252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.957994    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958031    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert podName:c82b956f-d109-4a4e-9a90-f298f5dcaf11 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.95802074 +0000 UTC m=+37.681366066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jgczn" (UID: "c82b956f-d109-4a4e-9a90-f298f5dcaf11") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:03.957994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958056    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.958047051 +0000 UTC m=+37.681392393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:03.958076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958114    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958143    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.958133598 +0000 UTC m=+37.681478926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : secret "router-metrics-certs-default" not found
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958174    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:03.958639 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958215    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:03.959397 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958226    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls podName:8b210005-d749-4ec8-ad41-bc7afaeb7011 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.95821595 +0000 UTC m=+37.681561289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qrsxv" (UID: "8b210005-d749-4ec8-ad41-bc7afaeb7011") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:03.959397 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958227    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:03.959397 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958258    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.958249602 +0000 UTC m=+37.681594942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:03.959397 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958285    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:03.959397 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958294    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:03.959397 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958321    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.958312937 +0000 UTC m=+37.681658266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:03.959397 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958319    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:03.959397 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:03.958389    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.958370024 +0000 UTC m=+37.681715353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:04.064981 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.064943    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh" event={"ID":"83e1b5fd-c06c-45a2-9c74-f29515e920ff","Type":"ContainerStarted","Data":"38ecf159da94dd364759730079075466d516dda99e3f7428767380e98dd759c4"}
Oct 31 07:29:04.462568 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.462528    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:04.476876 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.476842    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:04.548269 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.548232    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:04.563555 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.563519    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:04.563722 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.563578    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:04.563889 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:04.563861    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:04.564029 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:04.563938    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:36.563918148 +0000 UTC m=+66.287263506 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : secret "metrics-daemon-secret" not found
Oct 31 07:29:04.572787 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.572750    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:04.797898 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.797811    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:04.896433 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.896366    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh" podStartSLOduration=389.742784121 podStartE2EDuration="6m31.896344202s" totalImagesPullingTime="2.153560081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:04.083087992 +0000 UTC m=+33.806433338" watchObservedRunningTime="2026-10-31 07:29:04.896344202 +0000 UTC m=+34.619689549"
Oct 31 07:29:04.896823 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:04.896799    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l4tmf"]
Oct 31 07:29:04.934619 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:04.934577    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9420864f_c0f7_47d3_ad36_b4b15ba44b57.slice/crio-5b9b500794c02c5d2cf87664a4918983691db8c4a1877b585366ff878b09836d WatchSource:0}: Error finding container 5b9b500794c02c5d2cf87664a4918983691db8c4a1877b585366ff878b09836d: Status 404 returned error can't find the container with id 5b9b500794c02c5d2cf87664a4918983691db8c4a1877b585366ff878b09836d
Oct 31 07:29:05.069490 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:05.069383    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l4tmf" event={"ID":"9420864f-c0f7-47d3-ad36-b4b15ba44b57","Type":"ContainerStarted","Data":"5b9b500794c02c5d2cf87664a4918983691db8c4a1877b585366ff878b09836d"}
Oct 31 07:29:05.122927 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:05.122897    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d5t72"]
Oct 31 07:29:05.276268 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:05.276230    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686827ed_5eb8_4cdc_aeca_176b015eb4f8.slice/crio-b334cc978c9906775e8b63073b308142eb2e79df23804556783b7d58a3756715 WatchSource:0}: Error finding container b334cc978c9906775e8b63073b308142eb2e79df23804556783b7d58a3756715: Status 404 returned error can't find the container with id b334cc978c9906775e8b63073b308142eb2e79df23804556783b7d58a3756715
Oct 31 07:29:06.075390 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.075328    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" event={"ID":"9a1a70f5-55db-4e36-959d-95bae0872fea","Type":"ContainerStarted","Data":"75766ab24dcf9762882b3c0d69a9d5f0641229d94501bd60c02eb5e5b8e2497a"}
Oct 31 07:29:06.077358 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.077328    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:06.078103 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.077936    2572 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-7h5mf container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body=
Oct 31 07:29:06.078103 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.077989    2572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" podUID="9a1a70f5-55db-4e36-959d-95bae0872fea" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused"
Oct 31 07:29:06.081567 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.081540    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d5t72" event={"ID":"686827ed-5eb8-4cdc-aeca-176b015eb4f8","Type":"ContainerStarted","Data":"b334cc978c9906775e8b63073b308142eb2e79df23804556783b7d58a3756715"}
Oct 31 07:29:06.084002 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.083978    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l" event={"ID":"87fbf516-95a9-4c2c-a41f-1a053ff98e6b","Type":"ContainerStarted","Data":"ce3afbad417df8792f912ef50b5b4eca9e75ad3a6e45dc06ca490b90541e3b8b"}
Oct 31 07:29:06.086322 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.086301    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rjjcf" event={"ID":"e684ad1d-ea28-4d75-aca0-8c12227ffada","Type":"ContainerStarted","Data":"42acaae64f1e406d56d97558dea6441f6cb158c06f6b2bd946f1fb8b0b30c1a2"}
Oct 31 07:29:06.088554 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.088528    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb" event={"ID":"59cf0458-de23-4a5e-884f-4a583e78ff38","Type":"ContainerStarted","Data":"bc9f81c3760a4a9f7875d843060747cfe9f48795c890a367ca869fc5822a5069"}
Oct 31 07:29:06.094759 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.094702    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" podStartSLOduration=388.175721198 podStartE2EDuration="6m33.09468523s" totalImagesPullingTime="4.918964032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:06.093080112 +0000 UTC m=+35.816425483" watchObservedRunningTime="2026-10-31 07:29:06.09468523 +0000 UTC m=+35.818030577"
Oct 31 07:29:06.113692 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.113632    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l" podStartSLOduration=388.152803386 podStartE2EDuration="6m33.113618861s" totalImagesPullingTime="4.960815475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:06.113505397 +0000 UTC m=+35.836850745" watchObservedRunningTime="2026-10-31 07:29:06.113618861 +0000 UTC m=+35.836964207"
Oct 31 07:29:06.138078 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.138022    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-rjjcf" podStartSLOduration=388.09858283 podStartE2EDuration="6m33.138004573s" totalImagesPullingTime="5.039421743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:06.136241098 +0000 UTC m=+35.859586470" watchObservedRunningTime="2026-10-31 07:29:06.138004573 +0000 UTC m=+35.861349924"
Oct 31 07:29:06.642383 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.642323    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb" podStartSLOduration=388.583132546 podStartE2EDuration="6m33.642302625s" totalImagesPullingTime="5.059170079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:06.155933442 +0000 UTC m=+35.879278815" watchObservedRunningTime="2026-10-31 07:29:06.642302625 +0000 UTC m=+36.365647973"
Oct 31 07:29:06.643953 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.643932    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cdwzx"]
Oct 31 07:29:06.670011 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.669898    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cdwzx"]
Oct 31 07:29:06.670358 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.670324    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.686653 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.686554    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e85bb8-780b-4540-8e9e-19990f22f515-data-volume\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.686653 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.686605    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.686844 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.686712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0e85bb8-780b-4540-8e9e-19990f22f515-crio-socket\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.686844 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.686752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.686844 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.686768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwg8\" (UniqueName: \"kubernetes.io/projected/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-api-access-mlwg8\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.759557 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.759042    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 07:29:06.760095 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.759911    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 07:29:06.760095 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.760062    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-frjtn\""
Oct 31 07:29:06.787529 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.787468    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e85bb8-780b-4540-8e9e-19990f22f515-data-volume\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.787697 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.787596    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.787697 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.787665    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0e85bb8-780b-4540-8e9e-19990f22f515-crio-socket\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.787806 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.787705    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.787806 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.787729    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwg8\" (UniqueName: \"kubernetes.io/projected/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-api-access-mlwg8\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.788433 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:06.788002    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:06.788433 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:06.788072    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.288054457 +0000 UTC m=+37.011399801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:06.788433 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.788245    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e85bb8-780b-4540-8e9e-19990f22f515-data-volume\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.788433 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.788318    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0e85bb8-780b-4540-8e9e-19990f22f515-crio-socket\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:06.788433 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:06.788359    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:06.788433 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:06.788406    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.288391601 +0000 UTC m=+37.011736940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:06.797425 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:06.797370    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwg8\" (UniqueName: \"kubernetes.io/projected/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-api-access-mlwg8\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:07.097028 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.096952    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/0.log"
Oct 31 07:29:07.097028 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.097001    2572 generic.go:356] "Generic (PLEG): container finished" podID="9a1a70f5-55db-4e36-959d-95bae0872fea" containerID="75766ab24dcf9762882b3c0d69a9d5f0641229d94501bd60c02eb5e5b8e2497a" exitCode=255
Oct 31 07:29:07.097551 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.097078    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" event={"ID":"9a1a70f5-55db-4e36-959d-95bae0872fea","Type":"ContainerDied","Data":"75766ab24dcf9762882b3c0d69a9d5f0641229d94501bd60c02eb5e5b8e2497a"}
Oct 31 07:29:07.097654 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.097629    2572 scope.go:122] "RemoveContainer" containerID="75766ab24dcf9762882b3c0d69a9d5f0641229d94501bd60c02eb5e5b8e2497a"
Oct 31 07:29:07.104854 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.103526    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"]
Oct 31 07:29:07.124628 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.124460    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"]
Oct 31 07:29:07.126263 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.124541    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"
Oct 31 07:29:07.126597 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.126453    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg" event={"ID":"c64dab3d-c1ed-4828-b71f-d80b1f7fab00","Type":"ContainerStarted","Data":"89de0ed0ac2ee3b0ad6366458989fe79a98f233a46589d39042dfc3fa3226cee"}
Oct 31 07:29:07.126597 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.126562    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w9f8b" event={"ID":"63139743-194a-4517-9eba-62a858a3b649","Type":"ContainerStarted","Data":"2b6fde05da65ec83d9fe3d034a3d2ec6b8b108fa7cf5ee12ec84323108a57874"}
Oct 31 07:29:07.158937 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.158849    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w9f8b" podStartSLOduration=1.6187744450000001 podStartE2EDuration="7.158813243s" totalImagesPullingTime="5.540038798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:07.157248542 +0000 UTC m=+36.880593897" watchObservedRunningTime="2026-10-31 07:29:07.158813243 +0000 UTC m=+36.882158592"
Oct 31 07:29:07.184449 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.184155    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg" podStartSLOduration=385.294271508 podStartE2EDuration="6m31.184133881s" totalImagesPullingTime="5.889862373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:07.182561543 +0000 UTC m=+36.905906891" watchObservedRunningTime="2026-10-31 07:29:07.184133881 +0000 UTC m=+36.907479234"
Oct 31 07:29:07.195130 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.192088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5nt\" (UniqueName: \"kubernetes.io/projected/12778afe-1cd7-4829-b2c5-ddf2f1232ac2-kube-api-access-5p5nt\") pod \"migrator-66df5d496b-tbdww\" (UID: \"12778afe-1cd7-4829-b2c5-ddf2f1232ac2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"
Oct 31 07:29:07.207001 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.206733    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.207001 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.206806    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.207759 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.207592    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g74pl\""
Oct 31 07:29:07.293062 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.293026    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:07.293218 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:07.293178    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:07.293302 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.293233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:07.293302 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:07.293291    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.293268519 +0000 UTC m=+38.016613859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:07.293423 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.293319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5nt\" (UniqueName: \"kubernetes.io/projected/12778afe-1cd7-4829-b2c5-ddf2f1232ac2-kube-api-access-5p5nt\") pod \"migrator-66df5d496b-tbdww\" (UID: \"12778afe-1cd7-4829-b2c5-ddf2f1232ac2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"
Oct 31 07:29:07.293423 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:07.293361    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:07.293423 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:07.293414    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.293395245 +0000 UTC m=+38.016740569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:07.304594 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.304540    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5nt\" (UniqueName: \"kubernetes.io/projected/12778afe-1cd7-4829-b2c5-ddf2f1232ac2-kube-api-access-5p5nt\") pod \"migrator-66df5d496b-tbdww\" (UID: \"12778afe-1cd7-4829-b2c5-ddf2f1232ac2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"
Oct 31 07:29:07.498658 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.498625    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"
Oct 31 07:29:07.807967 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:07.807898    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"]
Oct 31 07:29:07.812222 ip-10-0-130-121 kubenswrapper[2572]: W1031 07:29:07.811562    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12778afe_1cd7_4829_b2c5_ddf2f1232ac2.slice/crio-e0d83eec7e5abe19a895950e34e1bd7fce3e06ba02da5672135a0a7b3ae9758d WatchSource:0}: Error finding container e0d83eec7e5abe19a895950e34e1bd7fce3e06ba02da5672135a0a7b3ae9758d: Status 404 returned error can't find the container with id e0d83eec7e5abe19a895950e34e1bd7fce3e06ba02da5672135a0a7b3ae9758d
Oct 31 07:29:08.000990 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.000946    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:08.001170 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.001038    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:08.001170 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.001072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:08.001170 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.001105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:08.001371 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001165    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:08.001371 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001203    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:08.001371 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.001244    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:08.001371 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001270    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.001250463 +0000 UTC m=+45.724595805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:08.001371 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.001319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:08.001623 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.001391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:08.001623 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.001445    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:08.001714 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001638    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:08.001714 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001680    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:08.001810 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001776    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:08.001810 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001784    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:08.001810 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001688    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.001673464 +0000 UTC m=+45.725018802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : secret "router-metrics-certs-default" not found
Oct 31 07:29:08.001969 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001826    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert podName:c82b956f-d109-4a4e-9a90-f298f5dcaf11 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.001811229 +0000 UTC m=+45.725156553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jgczn" (UID: "c82b956f-d109-4a4e-9a90-f298f5dcaf11") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:08.001969 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001841    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.001832354 +0000 UTC m=+45.725177684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:08.001969 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001856    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.001847757 +0000 UTC m=+45.725193126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:08.001969 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001901    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:08.001969 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001929    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.001923275 +0000 UTC m=+45.725268612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:08.001969 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001902    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:08.001969 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001953    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls podName:8b210005-d749-4ec8-ad41-bc7afaeb7011 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.001947893 +0000 UTC m=+45.725293265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qrsxv" (UID: "8b210005-d749-4ec8-ad41-bc7afaeb7011") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:08.001969 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.001966    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:08.002290 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.002006    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.001993453 +0000 UTC m=+45.725338794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:08.113748 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.113678    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/1.log"
Oct 31 07:29:08.114421 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.114398    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/0.log"
Oct 31 07:29:08.114513 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.114444    2572 generic.go:356] "Generic (PLEG): container finished" podID="9a1a70f5-55db-4e36-959d-95bae0872fea" containerID="6914b5364454241da8b1f446be10e10463a4e06ef29c4c49f44dc9c8d5db401e" exitCode=255
Oct 31 07:29:08.114581 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.114525    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" event={"ID":"9a1a70f5-55db-4e36-959d-95bae0872fea","Type":"ContainerDied","Data":"6914b5364454241da8b1f446be10e10463a4e06ef29c4c49f44dc9c8d5db401e"}
Oct 31 07:29:08.114581 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.114572    2572 scope.go:122] "RemoveContainer" containerID="75766ab24dcf9762882b3c0d69a9d5f0641229d94501bd60c02eb5e5b8e2497a"
Oct 31 07:29:08.114834 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.114815    2572 scope.go:122] "RemoveContainer" containerID="6914b5364454241da8b1f446be10e10463a4e06ef29c4c49f44dc9c8d5db401e"
Oct 31 07:29:08.115055 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.115029    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7h5mf_openshift-console-operator(9a1a70f5-55db-4e36-959d-95bae0872fea)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" podUID="9a1a70f5-55db-4e36-959d-95bae0872fea"
Oct 31 07:29:08.116920 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.116532    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww" event={"ID":"12778afe-1cd7-4829-b2c5-ddf2f1232ac2","Type":"ContainerStarted","Data":"e0d83eec7e5abe19a895950e34e1bd7fce3e06ba02da5672135a0a7b3ae9758d"}
Oct 31 07:29:08.305260 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.305218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:08.305451 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:08.305318    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:08.305515 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.305498    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:08.305588 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.305578    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.305559174 +0000 UTC m=+40.028904503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:08.306009 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.305983    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:08.306117 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:08.306037    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.306022579 +0000 UTC m=+40.029367908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:09.122426 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:09.122394    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/1.log"
Oct 31 07:29:09.122821 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:09.122739    2572 scope.go:122] "RemoveContainer" containerID="6914b5364454241da8b1f446be10e10463a4e06ef29c4c49f44dc9c8d5db401e"
Oct 31 07:29:09.122920 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:09.122901    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7h5mf_openshift-console-operator(9a1a70f5-55db-4e36-959d-95bae0872fea)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" podUID="9a1a70f5-55db-4e36-959d-95bae0872fea"
Oct 31 07:29:09.124990 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:09.124959    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d5t72" event={"ID":"686827ed-5eb8-4cdc-aeca-176b015eb4f8","Type":"ContainerStarted","Data":"8f23e62399dd1cf7ac78c240cad88c94ff0fd97fd35cd987cb2ee88d968c7198"}
Oct 31 07:29:09.125100 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:09.125090    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:09.157929 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:09.157873    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d5t72" podStartSLOduration=36.241857627 podStartE2EDuration="39.157858764s" totalImagesPullingTime="2.916001137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:09.156595308 +0000 UTC m=+38.879940653" watchObservedRunningTime="2026-10-31 07:29:09.157858764 +0000 UTC m=+38.881204112"
Oct 31 07:29:10.130398 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:10.130360    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww" event={"ID":"12778afe-1cd7-4829-b2c5-ddf2f1232ac2","Type":"ContainerStarted","Data":"4ce8bf838da2cfbbb7ab1dd5aefccb083661ec2194ba0243b03e3d4cf9678012"}
Oct 31 07:29:10.148160 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:10.148106    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww" podStartSLOduration=1.328519406 podStartE2EDuration="3.148091377s" totalImagesPullingTime="1.819571971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:10.14551724 +0000 UTC m=+39.868862606" watchObservedRunningTime="2026-10-31 07:29:10.148091377 +0000 UTC m=+39.871436774"
Oct 31 07:29:10.326488 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:10.326445    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:10.326663 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:10.326552    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:10.326663 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:10.326600    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:10.326757 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:10.326674    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.32665891 +0000 UTC m=+44.050004261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:10.326757 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:10.326681    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:10.326757 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:10.326736    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.326722675 +0000 UTC m=+44.050068013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:10.525253 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:10.525221    2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:10.525623 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:10.525608    2572 scope.go:122] "RemoveContainer" containerID="6914b5364454241da8b1f446be10e10463a4e06ef29c4c49f44dc9c8d5db401e"
Oct 31 07:29:10.525815 ip-10-0-130-121 kubenswrapper[2572]: E1031 07:29:10.525797    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7h5mf_openshift-console-operator(9a1a70f5-55db-4e36-959d-95bae0872fea)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" podUID="9a1a70f5-55db-4e36-959d-95bae0872fea"
Oct 31 07:29:11.135167 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:11.135134    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l4tmf" event={"ID":"9420864f-c0f7-47d3-ad36-b4b15ba44b57","Type":"ContainerStarted","Data":"886e1736f15371fcd38810d5a2a24fe75d69dab8d74ac90a0a7cd40457237619"}
Oct 31 07:29:11.152748 ip-10-0-130-121 kubenswrapper[2572]: I1031 07:29:11.152700    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l4tmf" podStartSLOduration=34.17293586 podStartE2EDuration="40.152687031s" totalImagesPullingTime="5.979751171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:11.151807868 +0000 UTC m=+40.875153216" watchObservedRunningTime="2026-10-31 07:29:11.152687031 +0000 UTC m=+40.876032376"
Oct 31 07:29:11.223062 ip-10-0-130-121 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 07:29:11.232576 ip-10-0-130-121 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 07:29:11.232745 ip-10-0-130-121 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 07:29:11.232792 ip-10-0-130-121 systemd[1]: kubelet.service: Consumed 3.035s CPU time.
Oct 31 07:29:11.250687 ip-10-0-130-121 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 07:29:11.339375 ip-10-0-130-121 kubenswrapper[4913]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:11.339375 ip-10-0-130-121 kubenswrapper[4913]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 07:29:11.339375 ip-10-0-130-121 kubenswrapper[4913]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:11.339375 ip-10-0-130-121 kubenswrapper[4913]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:11.341415 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341331    4913 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:11.341415 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341416    4913 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341421    4913 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341425    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341428    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341432    4913 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341434    4913 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341438    4913 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341451    4913 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341454    4913 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341457    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341460    4913 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341462    4913 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341465    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341468    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341471    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341474    4913 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341476    4913 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341479    4913 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341482    4913 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341484    4913 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:11.341492 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341487    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341490    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341492    4913 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341496    4913 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341499    4913 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341502    4913 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341505    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341508    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341511    4913 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341513    4913 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341516    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341519    4913 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341522    4913 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341524    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341529    4913 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341533    4913 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341536    4913 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341539    4913 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341542    4913 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:11.341993 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341545    4913 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341548    4913 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341550    4913 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341553    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341556    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341558    4913 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341561    4913 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341563    4913 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341566    4913 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341570    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341573    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341576    4913 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341579    4913 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341581    4913 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341584    4913 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341587    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341590    4913 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341592    4913 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341595    4913 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341598    4913 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:11.342473 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341601    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341604    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341607    4913 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341610    4913 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341612    4913 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341615    4913 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341619    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341621    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341624    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341627    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341629    4913 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341632    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341635    4913 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341638    4913 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341640    4913 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341643    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341645    4913 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341648    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341650    4913 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:11.342969 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341654    4913 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341657    4913 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341660    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341662    4913 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341665    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341668    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341670    4913 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341673    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341676    4913 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341678    4913 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341681    4913 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341684    4913 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341687    4913 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341689    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341692    4913 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341694    4913 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341699    4913 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341703    4913 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341706    4913 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:11.343428 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341709    4913 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341711    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341714    4913 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341717    4913 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341720    4913 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341722    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341725    4913 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.341728    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342493    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342501    4913 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342504    4913 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342508    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342510    4913 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342513    4913 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342517    4913 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342520    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342523    4913 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342525    4913 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342529    4913 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342532    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:11.343913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342534    4913 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342537    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342540    4913 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342543    4913 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342546    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342548    4913 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342551    4913 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342554    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342556    4913 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342559    4913 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342562    4913 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342565    4913 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342567    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342570    4913 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342573    4913 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342576    4913 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342579    4913 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342582    4913 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342584    4913 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342587    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:11.344624 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342590    4913 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342593    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342596    4913 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342598    4913 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342601    4913 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342604    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342606    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342609    4913 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342612    4913 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342614    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342618    4913 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342620    4913 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342624    4913 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342627    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342629    4913 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342633    4913 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342635    4913 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342638    4913 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342641    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342644    4913 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:11.345174 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342646    4913 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342649    4913 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342652    4913 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342654    4913 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342658    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342661    4913 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342664    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342667    4913 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342669    4913 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342672    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342674    4913 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342677    4913 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342680    4913 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342683    4913 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342686    4913 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342689    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342692    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342695    4913 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342698    4913 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:11.345708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342701    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342704    4913 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342707    4913 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342710    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342712    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342715    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342718    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342722    4913 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342726    4913 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342730    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342732    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342735    4913 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342738    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342740    4913 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342743    4913 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342746    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342749    4913 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342752    4913 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342754    4913 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:11.346167 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342757    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342760    4913 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342762    4913 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342765    4913 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342768    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342771    4913 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342775    4913 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342779    4913 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342783    4913 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342787    4913 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342790    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342793    4913 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342795    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342800    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342802    4913 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.342805    4913 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343305    4913 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343311    4913 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343315    4913 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:11.346642 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343318    4913 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343321    4913 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343323    4913 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343326    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343329    4913 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343332    4913 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343335    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343338    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343341    4913 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343344    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343346    4913 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343354    4913 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343357    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343360    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343363    4913 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343365    4913 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343368    4913 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343371    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343373    4913 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343376    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:11.347124 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343379    4913 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343382    4913 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343385    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343388    4913 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343391    4913 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343393    4913 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343396    4913 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343398    4913 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343401    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343404    4913 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343406    4913 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343409    4913 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343412    4913 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343415    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343418    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343420    4913 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343423    4913 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343427    4913 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343432    4913 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:11.347608 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343436    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343439    4913 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343442    4913 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343445    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343449    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343452    4913 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343455    4913 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343457    4913 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343460    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343463    4913 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343466    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343468    4913 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343471    4913 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343473    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343477    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343480    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343483    4913 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343485    4913 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343488    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343490    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:11.348070 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343493    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343496    4913 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343499    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343502    4913 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343505    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343507    4913 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343510    4913 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343513    4913 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343515    4913 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343518    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343521    4913 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343524    4913 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343526    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343529    4913 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343532    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343534    4913 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343539    4913 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343542    4913 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343544    4913 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:11.348554 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343547    4913 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343550    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343552    4913 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343555    4913 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343558    4913 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343561    4913 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343563    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343566    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343569    4913 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343572    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343575    4913 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343577    4913 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343580    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343583    4913 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343586    4913 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343588    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343591    4913 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343593    4913 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343596    4913 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343599    4913 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:11.348998 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343601    4913 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343605    4913 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343607    4913 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343610    4913 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.343613    4913 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343704    4913 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343712    4913 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343718    4913 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343723    4913 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343729    4913 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343733    4913 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343737    4913 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343741    4913 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343744    4913 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343748    4913 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343752    4913 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343756    4913 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343759    4913 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343762    4913 flags.go:64] FLAG: --cgroup-root=""
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343766    4913 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343769    4913 flags.go:64] FLAG: --client-ca-file=""
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343772    4913 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343775    4913 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343780    4913 flags.go:64] FLAG: --cluster-domain=""
Oct 31 07:29:11.349708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343783    4913 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343786    4913 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343789    4913 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343793    4913 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343796    4913 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343800    4913 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343803    4913 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343806    4913 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343809    4913 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343812    4913 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343816    4913 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343819    4913 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343822    4913 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343826    4913 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343829    4913 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343833    4913 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343836    4913 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343841    4913 flags.go:64] FLAG: --enable-server="true"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343844    4913 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343849    4913 flags.go:64] FLAG: --event-burst="100"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343852    4913 flags.go:64] FLAG: --event-qps="50"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343856    4913 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343859    4913 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343862    4913 flags.go:64] FLAG: --eviction-hard=""
Oct 31 07:29:11.350540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343867    4913 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343870    4913 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343873    4913 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343876    4913 flags.go:64] FLAG: --eviction-soft=""
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343879    4913 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343882    4913 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343885    4913 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343889    4913 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343892    4913 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343895    4913 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343898    4913 flags.go:64] FLAG: --feature-gates=""
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343905    4913 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343908    4913 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343912    4913 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343915    4913 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343918    4913 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343922    4913 flags.go:64] FLAG: --help="false"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343925    4913 flags.go:64] FLAG: --hostname-override="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343928    4913 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343931    4913 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343935    4913 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343939    4913 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343943    4913 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 07:29:11.351142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343947    4913 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343950    4913 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343954    4913 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343957    4913 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343960    4913 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343964    4913 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343967    4913 flags.go:64] FLAG: --kube-reserved=""
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343970    4913 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343973    4913 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343976    4913 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343980    4913 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343983    4913 flags.go:64] FLAG: --lock-file=""
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343986    4913 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343989    4913 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343992    4913 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.343997    4913 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344000    4913 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344004    4913 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344007    4913 flags.go:64] FLAG: --logging-format="text"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344010    4913 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344014    4913 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344017    4913 flags.go:64] FLAG: --manifest-url=""
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344020    4913 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344024    4913 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344027    4913 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 07:29:11.351748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344031    4913 flags.go:64] FLAG: --max-pods="110"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344035    4913 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344038    4913 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344041    4913 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344044    4913 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344048    4913 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344051    4913 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344054    4913 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344061    4913 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344065    4913 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344068    4913 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344072    4913 flags.go:64] FLAG: --pod-cidr=""
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344075    4913 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344078    4913 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344081    4913 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344084    4913 flags.go:64] FLAG: --port="10250"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344087    4913 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344090    4913 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-022935f6e06f361fa"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344094    4913 flags.go:64] FLAG: --qos-reserved=""
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344097    4913 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344100    4913 flags.go:64] FLAG: --register-node="true"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344103    4913 flags.go:64] FLAG: --register-with-taints=""
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344107    4913 flags.go:64] FLAG: --registry-burst="10"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344110    4913 flags.go:64] FLAG: --registry-qps="5"
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344113    4913 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344116    4913 flags.go:64] FLAG: --reserved-memory=""
Oct 31 07:29:11.352588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344120    4913 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344124    4913 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344127    4913 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344130    4913 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344133    4913 flags.go:64] FLAG: --runonce="false"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344137    4913 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344140    4913 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344143    4913 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344146    4913 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344149    4913 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344152    4913 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344155    4913 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344159    4913 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344162    4913 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344165    4913 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344169    4913 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344172    4913 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344194    4913 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344199    4913 flags.go:64] FLAG: --system-cgroups=""
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344202    4913 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344207    4913 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344210    4913 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344213    4913 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344217    4913 flags.go:64] FLAG: --tls-min-version=""
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344220    4913 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 07:29:11.353225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344223    4913 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344226    4913 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344229    4913 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344232    4913 flags.go:64] FLAG: --v="2"
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344237    4913 flags.go:64] FLAG: --version="false"
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344241    4913 flags.go:64] FLAG: --vmodule=""
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344245    4913 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344250    4913 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344385    4913 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344389    4913 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344392    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344395    4913 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344398    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344401    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344404    4913 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344406    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344409    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344412    4913 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344414    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344417    4913 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344420    4913 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344422    4913 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:11.353817 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344427    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344430    4913 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344433    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344436    4913 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344439    4913 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344441    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344444    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344447    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344449    4913 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344452    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344455    4913 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344457    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344460    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344463    4913 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344465    4913 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344468    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344471    4913 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344474    4913 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344478    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:11.354400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344482    4913 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344485    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344488    4913 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344491    4913 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344495    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344497    4913 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344500    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344503    4913 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344506    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344508    4913 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344511    4913 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344514    4913 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344516    4913 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344520    4913 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344523    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344526    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344529    4913 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344532    4913 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344535    4913 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344537    4913 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:11.354979 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344540    4913 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344543    4913 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344545    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344548    4913 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344551    4913 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344553    4913 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344556    4913 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344558    4913 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344562    4913 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344565    4913 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344568    4913 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344570    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344573    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344576    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344579    4913 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344582    4913 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344584    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344587    4913 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344590    4913 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:11.355550 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344592    4913 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344595    4913 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344598    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344600    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344603    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344606    4913 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344610    4913 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344613    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344615    4913 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344618    4913 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344621    4913 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344624    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344627    4913 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344630    4913 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344632    4913 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344635    4913 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344638    4913 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344641    4913 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344643    4913 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344646    4913 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:11.356400 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344648    4913 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344652    4913 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344690    4913 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344694    4913 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344696    4913 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344699    4913 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344703    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344706    4913 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344708    4913 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344711    4913 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344715    4913 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344719    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344722    4913 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344725    4913 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.344752    4913 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:11.357208 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344787    4913 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344792    4913 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344795    4913 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344799    4913 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344802    4913 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344805    4913 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344808    4913 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344811    4913 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344813    4913 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344816    4913 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344819    4913 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344821    4913 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344824    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344827    4913 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344830    4913 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344832    4913 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344836    4913 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344839    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344843    4913 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:11.357730 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344847    4913 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344850    4913 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344852    4913 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344855    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344858    4913 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344861    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344864    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344867    4913 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344869    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344872    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344875    4913 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344877    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344880    4913 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344883    4913 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344885    4913 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344888    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344891    4913 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344894    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344896    4913 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344899    4913 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:11.358427 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344902    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344904    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344907    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344909    4913 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344912    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344915    4913 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344917    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344920    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344923    4913 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344925    4913 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344928    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344932    4913 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344935    4913 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344938    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344942    4913 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344945    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344948    4913 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344951    4913 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344954    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:11.358913 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344957    4913 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344959    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344962    4913 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344965    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344967    4913 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344970    4913 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344973    4913 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344975    4913 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344978    4913 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344980    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344983    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344986    4913 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344988    4913 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344991    4913 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344994    4913 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.344996    4913 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345000    4913 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345003    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345006    4913 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345008    4913 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:11.359422 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345011    4913 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345014    4913 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345016    4913 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345019    4913 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345023    4913 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345025    4913 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345028    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345031    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345033    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345036    4913 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345039    4913 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345041    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345045    4913 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345048    4913 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345051    4913 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345053    4913 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345056    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345058    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345061    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:11.359900 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345064    4913 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345067    4913 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345069    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345072    4913 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345075    4913 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345078    4913 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345080    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345083    4913 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.345085    4913 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.348637    4913 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.348654    4913 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.348671    4913 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.348676    4913 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348731    4913 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348737    4913 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348741    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348744    4913 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348747    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348750    4913 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:11.360368 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348752    4913 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348755    4913 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348758    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348761    4913 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348764    4913 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348767    4913 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348770    4913 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348773    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348775    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348778    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348781    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348783    4913 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348786    4913 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348789    4913 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348792    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348794    4913 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348797    4913 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348800    4913 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348802    4913 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348805    4913 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:11.360805 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348808    4913 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348811    4913 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348814    4913 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348818    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348821    4913 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348823    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348826    4913 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348829    4913 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348831    4913 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348834    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348837    4913 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348840    4913 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348842    4913 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348845    4913 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348848    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348851    4913 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348853    4913 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348856    4913 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348858    4913 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:11.361291 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348861    4913 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348864    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348867    4913 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348870    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348873    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348875    4913 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348878    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348880    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348884    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348886    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348889    4913 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348892    4913 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348894    4913 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348897    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348899    4913 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348902    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348904    4913 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348910    4913 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348913    4913 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:11.361754 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348915    4913 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348919    4913 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348923    4913 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348927    4913 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348930    4913 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348933    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348936    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348939    4913 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348943    4913 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348947    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348950    4913 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348953    4913 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348956    4913 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348958    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348961    4913 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348964    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348966    4913 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348969    4913 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348971    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:11.362225 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348974    4913 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348977    4913 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348979    4913 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348982    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348984    4913 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.348998    4913 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349002    4913 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349005    4913 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349008    4913 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349010    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349013    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349016    4913 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349019    4913 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349022    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349025    4913 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349027    4913 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349030    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349033    4913 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349035    4913 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349039    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:11.362675 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349041    4913 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349044    4913 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349047    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.349074    4913 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349698    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349827    4913 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349834    4913 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349839    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349843    4913 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349848    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349852    4913 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349856    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349861    4913 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349866    4913 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349877    4913 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:11.363148 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349882    4913 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349886    4913 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349891    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349895    4913 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349899    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349904    4913 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349908    4913 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349912    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349916    4913 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349921    4913 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349926    4913 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349931    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349940    4913 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349945    4913 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349949    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349953    4913 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349957    4913 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349962    4913 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349966    4913 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349971    4913 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:11.363522 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349975    4913 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349980    4913 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349985    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349989    4913 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.349994    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350003    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350007    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350011    4913 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350066    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350072    4913 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350076    4913 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350080    4913 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350085    4913 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350090    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350096    4913 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350100    4913 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350106    4913 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350110    4913 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350115    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350119    4913 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:11.363995 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350130    4913 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350134    4913 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350139    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350143    4913 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350147    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350151    4913 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350155    4913 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350160    4913 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350164    4913 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350168    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350173    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350177    4913 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350203    4913 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350209    4913 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350214    4913 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350218    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350223    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350228    4913 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350236    4913 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:11.364540 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350243    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350248    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350253    4913 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350259    4913 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350263    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350274    4913 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350281    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350286    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350290    4913 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350294    4913 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350298    4913 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350302    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350306    4913 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350310    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350315    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350320    4913 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350325    4913 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350330    4913 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350349    4913 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350353    4913 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:11.365026 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350358    4913 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350445    4913 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350449    4913 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350454    4913 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350458    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350464    4913 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350469    4913 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350472    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350474    4913 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350477    4913 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350480    4913 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350482    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350485    4913 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350488    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350491    4913 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350494    4913 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:11.365581 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.350531    4913 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350571    4913 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350577    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350581    4913 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350584    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350587    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350590    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350593    4913 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350596    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350599    4913 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350604    4913 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350607    4913 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350610    4913 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350614    4913 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350620    4913 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350625    4913 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350629    4913 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350632    4913 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350634    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350637    4913 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:11.366004 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350640    4913 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350643    4913 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350646    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350648    4913 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350651    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350654    4913 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350657    4913 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350659    4913 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350662    4913 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350665    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350667    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350670    4913 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350673    4913 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350676    4913 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350679    4913 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350681    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350685    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350688    4913 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350692    4913 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:11.366820 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350696    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350699    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350702    4913 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350705    4913 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350708    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350713    4913 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350717    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350720    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350723    4913 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350726    4913 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350728    4913 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350731    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350734    4913 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350736    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350739    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350742    4913 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350745    4913 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350748    4913 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350750    4913 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350753    4913 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:11.367446 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350756    4913 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350758    4913 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350761    4913 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350764    4913 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350766    4913 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350769    4913 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350772    4913 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350775    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350778    4913 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350781    4913 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350786    4913 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350790    4913 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350793    4913 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350795    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350798    4913 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350801    4913 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350804    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350806    4913 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350809    4913 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:11.367960 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350812    4913 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350814    4913 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350817    4913 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350820    4913 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350823    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350825    4913 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350828    4913 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350831    4913 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350835    4913 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350838    4913 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350840    4913 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350843    4913 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350845    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350848    4913 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350851    4913 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350853    4913 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350856    4913 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350859    4913 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350862    4913 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350865    4913 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:11.368437 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350867    4913 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350870    4913 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350872    4913 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350877    4913 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350880    4913 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350882    4913 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350885    4913 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350887    4913 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:11.350890    4913 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.350943    4913 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.351930    4913 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.351982    4913 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.352151    4913 server.go:1014] "Starting client certificate rotation"
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.352308    4913 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.352372    4913 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:23:30 +0000 UTC" deadline="2028-05-23 04:35:01.137642478 +0000 UTC"
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.352410    4913 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13677h5m49.785236306s"
Oct 31 07:29:11.368956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.352727    4913 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:29:11.369484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.353851    4913 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:29:11.369484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.355138    4913 log.go:25] "Validated CRI v1 runtime API"
Oct 31 07:29:11.369484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.357192    4913 log.go:25] "Validated CRI v1 image API"
Oct 31 07:29:11.369484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.357891    4913 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 07:29:11.369484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.359683    4913 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f3ffebdc-4b6e-4983-ad76-d443fe514e70:/dev/nvme0n1p4 fe4880a2-a519-45fa-81f8-3258dbd2f6f8:/dev/nvme0n1p3]
Oct 31 07:29:11.369712 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.359703    4913 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/064d02052d83429e3515ed4ca913ccbd9fa7652b571971250a6bd9b68cb83930/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/064d02052d83429e3515ed4ca913ccbd9fa7652b571971250a6bd9b68cb83930/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a219d65cff1ea71f8e7078396466c39bf362b51439e959c770cc7e98adf00ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a219d65cff1ea71f8e7078396466c39bf362b51439e959c770cc7e98adf00ec/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e67e7ae3b7a66c55a6ff6eba5d7769dfadfff9b2364ca4735a86ad49a5904ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e67e7ae3b7a66c55a6ff6eba5d7769dfadfff9b2364ca4735a86ad49a5904ce/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/249c027c5adfb01f21ba0fad290fd1b1767eb9b119bec5e0f2653d825f28660b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/249c027c5adfb01f21ba0fad290fd1b1767eb9b119bec5e0f2653d825f28660b/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e27b0d544e40ec8f0656b7c9c8c64a5d7178ffcd9dd6cbc0528e661514171bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e27b0d544e40ec8f0656b7c9c8c64a5d7178ffcd9dd6cbc0528e661514171bc/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f9822d2e207c346eccf487923220859228defbdb9a2878a100a2071ca3c7e18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f9822d2e207c346eccf487923220859228defbdb9a2878a100a2071ca3c7e18/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/422e06340d9787fb5b64f74789dea9c9a5ce8a523b043e8f1024a030ab174df2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/422e06340d9787fb5b64f74789dea9c9a5ce8a523b043e8f1024a030ab174df2/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/498f067da301d7bc89431b26856057279333647defa3ec6ca18e5d5c084384df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/498f067da301d7bc89431b26856057279333647defa3ec6ca18e5d5c084384df/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ae2672ff707f35394a0a8bfb496ed32c358570deaf62300c91b68bcbaba8863/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ae2672ff707f35394a0a8bfb496ed32c358570deaf62300c91b68bcbaba8863/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b9b500794c02c5d2cf87664a4918983691db8c4a1877b585366ff878b09836d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b9b500794c02c5d2cf87664a4918983691db8c4a1877b585366ff878b09836d/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62fb044edc2c1ec66daa935b2fadcf4f37117b386ba45dc02767f6c5fa474cd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62fb044edc2c1ec66daa935b2fadcf4f37117b386ba45dc02767f6c5fa474cd1/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/703cf2933db32a4c3b02047cff77e6f3dd07ff1341bb2f1702644270240c21c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/703cf2933db32a4c3b02047cff77e6f3dd07ff1341bb2f1702644270240c21c0/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fe1b3d7887719f095609272bea432bd413aef0b6ca9b72f0b3c1a8b8b2811b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fe1b3d7887719f095609272bea432bd413aef0b6ca9b72f0b3c1a8b8b2811b4/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d0f9e9079fc53a703379391f34ab9b1ebc69d7e349be659863c7e674037a4f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d0f9e9079fc53a703379391f34ab9b1ebc69d7e349be659863c7e674037a4f3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b14dfce64a90aa78f790e735f4a689603802b3ee22d0964bc8702bdd43c86beb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b14dfce64a90aa78f790e735f4a689603802b3ee22d0964bc8702bdd43c86beb/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b334cc978c9906775e8b63073b308142eb2e79df23804556783b7d58a3756715/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b334cc978c9906775e8b63073b308142eb2e79df23804556783b7d58a3756715/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b66fd7932ec7f3252e9d8c75e32c041e93be091da7d3dd022bb68bddb510b4dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b66fd7932ec7f3252e9d8c75e32c041e93be091da7d3dd022bb68bddb510b4dd/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd8b34bfe5082da35ca9028be98db54e34eed727be0429cef3850231413e6886/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd8b34bfe5082da35ca9028be98db54e34eed727be0429cef3850231413e6886/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d03de8ef1759a01a591b57d7b999191414c4e19d05d3db7610a8c67074a59fb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d03de8ef1759a01a591b57d7b999191414c4e19d05d3db7610a8c67074a59fb9/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0d83eec7e5abe19a895950e34e1bd7fce3e06ba02da5672135a0a7b3ae9758d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0d83eec7e5abe19a895950e34e1bd7fce3e06ba02da5672135a0a7b3ae9758d/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3e2b63ff27a4d5fab22a0ae75d10d10781aa0169521f6f236ce0748ef466767/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3e2b63ff27a4d5fab22a0ae75d10d10781aa0169521f6f236ce0748ef466767/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5b849564cdfc431c9a4da01bb4b99d3c869a00c462b30f2177a77cef3636660/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5b849564cdfc431c9a4da01bb4b99d3c869a00c462b30f2177a77cef3636660/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e975873444920ca169dec3ea811183ba6de514cd108e34a48d10ac21ae09856f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e975873444920ca169dec3ea811183ba6de514cd108e34a48d10ac21ae09856f/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~projected/kube-api-access-6vgww:{mountpoint:/var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~projected/kube-api-access-6vgww major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0546b914-d103-41a5-9e5b-13c6e41bcd85/volumes/kubernetes.io~projected/kube-api-access-xc79k:{mountpoint:/var/lib/kubelet/pods/0546b914-d103-41a5-9e5b-13c6e41bcd85/volumes/kubernetes.io~projected/kube-api-access-xc79k major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d264640-a684-42bd-8787-86d2382a7eb4/volumes/kubernetes.io~projected/kube-api-access-v5pdw:{mountpoint:/var/lib/kubelet/pods/0d264640-a684-42bd-8787-86d2382a7eb4/volumes/kubernetes.io~projected/kube-api-access-v5pdw major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12778afe-1cd7-4829-b2c5-ddf2f1232ac2/volumes/kubernetes.io~projected/kube-api-access-5p5nt:{mountpoint:/var/lib/kubelet/pods/12778afe-1cd7-4829-b2c5-ddf2f1232ac2/volumes/kubernetes.io~projected/kube-api-access-5p5nt major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1820478c-a566-4c78-b063-e8ceb9c6e221/volumes/kubernetes.io~projected/kube-api-access-cqf8w:{mountpoint:/var/lib/kubelet/pods/1820478c-a566-4c78-b063-e8ceb9c6e221/volumes/kubernetes.io~projected/kube-api-access-cqf8w major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3432e0f3-ae43-4815-a8b1-38e7b7c0222d/volumes/kubernetes.io~projected/kube-api-access-rjrfw:{mountpoint:/var/lib/kubelet/pods/3432e0f3-ae43-4815-a8b1-38e7b7c0222d/volumes/kubernetes.io~projected/kube-api-access-rjrfw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/385e4d20-8bd7-4933-8a7d-4359e0301d89/volumes/kubernetes.io~projected/kube-api-access-tmtlv:{mountpoint:/var/lib/kubelet/pods/385e4d20-8bd7-4933-8a7d-4359e0301d89/volumes/kubernetes.io~projected/kube-api-access-tmtlv major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3adfe0f6-05ab-44f9-8884-06c021c42f5a/volumes/kubernetes.io~projected/kube-api-access-2brvk:{mountpoint:/var/lib/kubelet/pods/3adfe0f6-05ab-44f9-8884-06c021c42f5a/volumes/kubernetes.io~projected/kube-api-access-2brvk major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4acf6e61-938d-495e-92ea-c1bc343b9406/volumes/kubernetes.io~projected/kube-api-access-r5nn6:{mountpoint:/var/lib/kubelet/pods/4acf6e61-938d-495e-92ea-c1bc343b9406/volumes/kubernetes.io~projected/kube-api-access-r5nn6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59cf0458-de23-4a5e-884f-4a583e78ff38/volumes/kubernetes.io~projected/kube-api-access-n4fk2:{mountpoint:/var/lib/kubelet/pods/59cf0458-de23-4a5e-884f-4a583e78ff38/volumes/kubernetes.io~projected/kube-api-access-n4fk2 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59cf0458-de23-4a5e-884f-4a583e78ff38/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/59cf0458-de23-4a5e-884f-4a583e78ff38/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b3d13df-dc50-41c8-a7cd-051fb9934a34/volumes/kubernetes.io~projected/kube-api-access-pc9xp:{mountpoint:/var/lib/kubelet/pods/5b3d13df-dc50-41c8-a7cd-051fb9934a34/volumes/kubernetes.io~projected/kube-api-access-pc9xp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca/volumes/kubernetes.io~projected/kube-api-access-m9fxd:{mountpoint:/var/lib/kubelet/pods/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca/volumes/kubernetes.io~projected/kube-api-access-m9fxd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6121aa17-a37d-47de-af76-08c2aa174568/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6121aa17-a37d-47de-af76-08c2aa174568/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63139743-194a-4517-9eba-62a858a3b649/volumes/kubernetes.io~projected/kube-api-access-d8d86:{mountpoint:/var/lib/kubelet/pods/63139743-194a-4517-9eba-62a858a3b649/volumes/kubernetes.io~projected/kube-api-access-d8d86 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63139743-194a-4517-9eba-62a858a3b649/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/63139743-194a-4517-9eba-62a858a3b649/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/686827ed-5eb8-4cdc-aeca-176b015eb4f8/volumes/kubernetes.io~projected/kube-api-access-j988n:{mountpoint:/var/lib/kubelet/pods/686827ed-5eb8-4cdc-aeca-176b015eb4f8/volumes/kubernetes.io~projected/kube-api-access-j988n major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volumes/kubernetes.io~projected/kube-api-access-mljlb:{mountpoint:/var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volumes/kubernetes.io~projected/kube-api-access-mljlb major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83e1b5fd-c06c-45a2-9c74-f29515e920ff/volumes/kubernetes.io~projected/kube-api-access-cg4vv:{mountpoint:/var/lib/kubelet/pods/83e1b5fd-c06c-45a2-9c74-f29515e920ff/volumes/kubernetes.io~projected/kube-api-access-cg4vv major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87fbf516-95a9-4c2c-a41f-1a053ff98e6b/volumes/kubernetes.io~projected/kube-api-access-hjlnk:{mountpoint:/var/lib/kubelet/pods/87fbf516-95a9-4c2c-a41f-1a053ff98e6b/volumes/kubernetes.io~projected/kube-api-access-hjlnk major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87fbf516-95a9-4c2c-a41f-1a053ff98e6b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/87fbf516-95a9-4c2c-a41f-1a053ff98e6b/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b210005-d749-4ec8-ad41-bc7afaeb7011/volumes/kubernetes.io~projected/kube-api-access-4wgq9:{mountpoint:/var/lib/kubelet/pods/8b210005-d749-4ec8-ad41-bc7afaeb7011/volumes/kubernetes.io~projected/kube-api-access-4wgq9 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~projected/kube-api-access-dw9xm:{mountpoint:/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~projected/kube-api-access-dw9xm major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~projected/kube-api-access-ln7bd:{mountpoint:/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~projected/kube-api-access-ln7bd major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9420864f-c0f7-47d3-ad36-b4b15ba44b57/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9420864f-c0f7-47d3-ad36-b4b15ba44b57/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a1a70f5-55db-4e36-959d-95bae0872fea/volumes/kubernetes.io~projected/kube-api-access-774lj:{mountpoint:/var/lib/kubelet/pods/9a1a70f5-55db-4e36-959d-95bae0872fea/volumes/kubernetes.io~projected/kube-api-access-774lj major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a1a70f5-55db-4e36-959d-95bae0872fea/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9a1a70f5-55db-4e36-959d-95bae0872fea/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a329ba42-e8f3-43ed-9355-d906a577e176/volumes/kubernetes.io~projected/kube-api-access-5xsdc:{mountpoint:/var/lib/kubelet/pods/a329ba42-e8f3-43ed-9355-d906a577e176/volumes/kubernetes.io~projected/kube-api-access-5xsdc major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0e85bb8-780b-4540-8e9e-19990f22f515/volumes/kubernetes.io~projected/kube-api-access-mlwg8:{mountpoint:/var/lib/kubelet/pods/b0e85bb8-780b-4540-8e9e-19990f22f515/volumes/kubernetes.io~projected/kube-api-access-mlwg8 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bca13e-12d2-4673-8223-92f7755eac90/volumes/kubernetes.io~projected/kube-api-access-zl6bn:{mountpoint:/var/lib/kubelet/pods/b5bca13e-12d2-4673-8223-92f7755eac90/volumes/kubernetes.io~projected/kube-api-access-zl6bn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~projected/kube-api-access-lcdmk:{mountpoint:/var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~projected/kube-api-access-lcdmk major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~secret/stats-auth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c64dab3d-c1ed-4828-b71f-d80b1f7fab00/volumes/kubernetes.io~projected/kube-api-access-m7dpq:{mountpoint:/var/lib/kubelet/pods/c64dab3d-c1ed-4828-b71f-d80b1f7fab00/volumes/kubernetes.io~projected/kube-api-access-m7dpq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e684ad1d-ea28-4d75-aca0-8c12227ffada/volumes/kubernetes.io~projected/kube-api-access-5hgs9:{mountpoint:/var/lib/kubelet/pods/e684ad1d-ea28-4d75-aca0-8c12227ffada/volumes/kubernetes.io~projected/kube-api-access-5hgs9 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e684ad1d-ea28-4d75-aca0-8c12227ffada/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e684ad1d-ea28-4d75-aca0-8c12227ffada/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff383e3f-6228-49ad-bf4e-f43d3eaadf03/volumes/kubernetes.io~projected/kube-api-access-vwmn5:{mountpoint:/var/lib/kubelet/pods/ff383e3f-6228-49ad-bf4e-f43d3eaadf03/volumes/kubernetes.io~projected/kube-api-access-vwmn5 major:0 minor:217 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/039429b09b76a6e50f528cda8c6617d75cda236ccdd68d4bd39106ef93344985/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/46e20780b7374b33e57e5785a47e19ac5bab8b016e41f1010c8ed1fe7b720ffc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/87bac8d7fa02b2c6921deb115282d2d4ed2736908b1a54ab8ab63a4e324250c2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a70ed248b62d624eaf2705d6c46cdf05384f461532bd3b38d957accbeaf8c92f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5042e9f1fe29816f9cc497b3d9d7b456e223116f02321b4a8caf22c5f2731200/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/30a14c98e5a8743f038daae5f6d346c34a87a5f19a93a26fd614171beb2949ed/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0527650e83ce563cc56d6c1ce572fa713b535c01fe3bc82fe6a81b1987286c4b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d061846b367663f53fbf4f644677e5506a7a29829f2741f767d972d434dbffd8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c997415e5b11ee2b3eb7c95108e68ecd03042415c144cd918953118eeada8bbe/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/20b7c81fc89f2bcbae9d6f75f9b6a0179f43442ca90a0d93b87ceb5153163fcf/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/c108b9aad01c81037f7dbf2753829e5ff05925bd494465e628c1197f9421e843/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/489273ce99b26a61a63b937fcb7c97532e9000436ed7a2a530e9a0a38ac58ec6/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6a21061ce5317ef2bfb603e6dc76dc7ab60b121495328f421ea611a6eff880ea/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/851d56d9c67a24ec59ace7bde06a9b51cc75ec28bb94001c53ae207b09730453/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2ca03121b9d6ecbde4ff741e685068b3f4976a06b812ee20d6886793e26c21b1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1ce84ef3c4f938745d175d87ebaeea750070a02f6e01e6268af69a1017d18a7a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bdf75836348cca9e0e3a4118742b0c040f72b63aa6d5a5138c24c366e31f4def/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/43ddacca2c1efccc2f3ee49e87e13e65f9e1b92e23d49448c72943fb5ff075f3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/99f9ae98024a959620a2904180c6b5cc92946bfeb9382cd2e6fbf5edce964f22/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4c26586136c48f3146e39c403e87027ca8cd0f84c3febf04898e7c3ea1813fdf/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4992186da70406332f6f457c47d626c2f55b4d1ed7391e789554313d5382121e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b41e1aa8bf9d55030510b68a04868fb039d5800c56c468e6ceafc6b5de556a23/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/52acd2fda84222567c4540d8ebc43e79a738371e2f9e65ab38b56de93259c859/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d6925d8b5315de710dec400b226e1836fc6d96d5ce4cdd6afae7ca94754bbc17/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a9089ee51ef6dea2f35d23ffe07a230194654d8937c99829ba51c4daac5d340a/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/24f5f2a2e8b1765870c3fc63bae616f84f1079e31fe2152b4fdb19ebe4089edf/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/eadfe9a33df7e5d251693ea3512c6646cca3a935ac76d153973f07f54766b98a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e476c0f87b34736bf30b2ec4e81d2da78faf1ccde807dddafa5d84306290bff4/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2f718be0f7696a86802d634126972abb9ae53b9b03ae0d9439a1b611fa954f38/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/41b1043fe9dd585e91cb8054e2a02a5e68f30da2a4236496a338015224b4b36e/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/d9e23881d7fe621607c64c69f622749efe09b1ad1a51e18f9201e475788f425d/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/3e32b10d546620d148d0cfc820480f02677afe9c26cbcff6d31bca24b6ee2de0/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/6932385cfa1bea54793144783b79017a55fc88c09551889dc64a9cdf3d614b6c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/f5676e44be4f45637bb1ecaf3944458861a6e95c7f32282e32eda395cf189185/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/ba133b08e244f1e2e242beca2f69ea36fd9337241cec720b7f3fc316de85aeda/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/289bef68b8097bd1045e1d98fe0a7477372e0da31af3e02a0746656bdc2a3446/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/417d5f44f8235fcf9f0902569abf03a21cb3d2a52060844f71af1b05e87ffee2/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/728f676188f778036a09535807d3b527d729f4d81f7da151ec66b9028dfdd2a5/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/be8a27f6bd455b3eaadd67f1865d73d69a025e502812e469d619789c19596d87/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/d0b148ef3d6eccca337e6bfc29d6cc623ec833afd8a6800b23f160d2fe804b5e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/fe2c422c04947538f1cc18fbdd98829a6f987a7698327c254b4682e869054813/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/a0b0e47f242e6382068188c088e36ffaa8eff962db3fcb2f2eeaea7788e37826/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/9a923a1c622becd5206ed173ece4a4196ecd3c269226cff894c079a7547a8045/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/9c79cabd2f9227aeb5dd08c0801c6bf7abbc2c3d2fca0d99cd35601666bfa8a1/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/db4d55735ac19223b2b26c5d20b627ea0dedfc3529caa7439c3748186c76e1f8/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/5f8909c1a63b793cc587d2ccff8ab8355cadb65143961cb0c6761c5d934c47a1/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/ec9fb6647ec5e4f71ebe212197e75cefa6cb58de7219236edfcdff392c1a9396/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0a46a198961516736f727f9bade8db13d70ed34b5066c4624614f68d573eb3f5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6d1a84417c41c86ad2784dd9f7fa09bdd764f70da89cfb771f561e887b73a823/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/75ed2bfc9a455960c82b9cfe16903f17f3f6f22925c9b115ecbeee99194700ff/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3f80b68e759c1dc6dda244e77d173b659ec6ce8ca479400d4e08c7cd8a1f0ac1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f0d1c85a8ef9b60c07f6024ea0b1501ae1183b1922ed83c4144ecea7e2cf5e21/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/a744fb5ebcb49fc9e66d32e56e46ba9649f0aa728b674abe20aec18545df2b65/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2130a0b4a897ea2517bc1ee384c6426496aa362e3d8e2175f4a3fbc091732f3e/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1231481a093325bca293a83a9a2e52d9744414690d24e13590cd66b457ee93c1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5a1b3e1a7392661301b22bf532997b8264a925a6c26e54a39562480f607492a5/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368128    4913 manager.go:217] Machine: {Timestamp:2026-10-31 07:29:11.367677409 +0000 UTC m=+0.057445692 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101077 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec219188a6970aae52828606e75c02c7 SystemUUID:ec219188-a697-0aae-5282-8606e75c02c7 BootID:cfefab99-c1c9-4fe0-862a-d5fe14dcd440 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b14dfce64a90aa78f790e735f4a689603802b3ee22d0964bc8702bdd43c86beb/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b0e85bb8-780b-4540-8e9e-19990f22f515/volumes/kubernetes.io~projected/kube-api-access-mlwg8 DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d264640-a684-42bd-8787-86d2382a7eb4/volumes/kubernetes.io~projected/kube-api-access-v5pdw DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b66fd7932ec7f3252e9d8c75e32c041e93be091da7d3dd022bb68bddb510b4dd/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12778afe-1cd7-4829-b2c5-ddf2f1232ac2/volumes/kubernetes.io~projected/kube-api-access-5p5nt DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/385e4d20-8bd7-4933-8a7d-4359e0301d89/volumes/kubernetes.io~projected/kube-api-access-tmtlv DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a1a70f5-55db-4e36-959d-95bae0872fea/volumes/kubernetes.io~projected/kube-api-access-774lj DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd8b34bfe5082da35ca9028be98db54e34eed727be0429cef3850231413e6886/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/498f067da301d7bc89431b26856057279333647defa3ec6ca18e5d5c084384df/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca/volumes/kubernetes.io~projected/kube-api-access-m9fxd DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~projected/kube-api-access-6vgww DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3e2b63ff27a4d5fab22a0ae75d10d10781aa0169521f6f236ce0748ef466767/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3432e0f3-ae43-4815-a8b1-38e7b7c0222d/volumes/kubernetes.io~projected/kube-api-access-rjrfw DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83e1b5fd-c06c-45a2-9c74-f29515e920ff/volumes/kubernetes.io~projected/kube-api-access-cg4vv DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e67e7ae3b7a66c55a6ff6eba5d7769dfadfff9b2364ca4735a86ad49a5904ce/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0d83eec7e5abe19a895950e34e1bd7fce3e06ba02da5672135a0a7b3ae9758d/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0546b914-d103-41a5-9e5b-13c6e41bcd85/volumes/kubernetes.io~projected/kube-api-access-xc79k DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6121aa17-a37d-47de-af76-08c2aa174568/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/249c027c5adfb01f21ba0fad290fd1b1767eb9b119bec5e0f2653d825f28660b/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b9b500794c02c5d2cf87664a4918983691db8c4a1877b585366ff878b09836d/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87fbf516-95a9-4c2c-a41f-1a053ff98e6b/volumes/kubernetes.io~projected/kube-api-access-hjlnk DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff383e3f-6228-49ad-bf4e-f43d3eaadf03/volumes/kubernetes.io~projected/kube-api-access-vwmn5 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e27b0d544e40ec8f0656b7c9c8c64a5d7178ffcd9dd6cbc0528e661514171bc/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f9822d2e207c346eccf487923220859228defbdb9a2878a100a2071ca3c7e18/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87fbf516-95a9-4c2c-a41f-1a053ff98e6b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~projected/kube-api-access-lcdmk DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4acf6e61-938d-495e-92ea-c1bc343b9406/volumes/kubernetes.io~projected/kube-api-access-r5nn6 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a219d65cff1ea71f8e7078396466c39bf362b51439e959c770cc7e98adf00ec/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5b849564cdfc431c9a4da01bb4b99d3c869a00c462b30f2177a77cef3636660/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd4df54c-65e4-4dba-943c-17219493fc36/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~projected/kube-api-access-dw9xm DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/703cf2933db32a4c3b02047cff77e6f3dd07ff1341bb2f1702644270240c21c0/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/064d02052d83429e3515ed4ca913ccbd9fa7652b571971250a6bd9b68cb83930/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59cf0458-de23-4a5e-884f-4a583e78ff38/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59cf0458-de23-4a5e-884f-4a583e78ff38/volumes/kubernetes.io~projected/kube-api-access-n4fk2 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e684ad1d-ea28-4d75-aca0-8c12227ffada/volumes/kubernetes.io~projected/kube-api-access-5hgs9 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/686827ed-5eb8-4cdc-aeca-176b015eb4f8/volumes/kubernetes.io~projected/kube-api-access-j988n DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b3d13df-dc50-41c8-a7cd-051fb9934a34/volumes/kubernetes.io~projected/kube-api-access-pc9xp DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bca13e-12d2-4673-8223-92f7755eac90/volumes/kubernetes.io~projected/kube-api-access-zl6bn DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e684ad1d-ea28-4d75-aca0-8c12227ffada/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e975873444920ca169dec3ea811183ba6de514cd108e34a48d10ac21ae09856f/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1820478c-a566-4c78-b063-e8ceb9c6e221/volumes/kubernetes.io~projected/kube-api-access-cqf8w DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63139743-194a-4517-9eba-62a858a3b649/volumes/kubernetes.io~projected/kube-api-access-d8d86 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/422e06340d9787fb5b64f74789dea9c9a5ce8a523b043e8f1024a030ab174df2/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62fb044edc2c1ec66daa935b2fadcf4f37117b386ba45dc02767f6c5fa474cd1/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b210005-d749-4ec8-ad41-bc7afaeb7011/volumes/kubernetes.io~projected/kube-api-access-4wgq9 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c64dab3d-c1ed-4828-b71f-d80b1f7fab00/volumes/kubernetes.io~projected/kube-api-access-m7dpq DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/021b18d4-0686-43c1-9faf-b1ca555d22a3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a329ba42-e8f3-43ed-9355-d906a577e176/volumes/kubernetes.io~projected/kube-api-access-5xsdc DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ae2672ff707f35394a0a8bfb496ed32c358570deaf62300c91b68bcbaba8863/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d03de8ef1759a01a591b57d7b999191414c4e19d05d3db7610a8c67074a59fb9/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~projected/kube-api-access-ln7bd DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9420864f-c0f7-47d3-ad36-b4b15ba44b57/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:272 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b334cc978c9906775e8b63073b308142eb2e79df23804556783b7d58a3756715/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63139743-194a-4517-9eba-62a858a3b649/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a1a70f5-55db-4e36-959d-95bae0872fea/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3adfe0f6-05ab-44f9-8884-06c021c42f5a/volumes/kubernetes.io~projected/kube-api-access-2brvk DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fe1b3d7887719f095609272bea432bd413aef0b6ca9b72f0b3c1a8b8b2811b4/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volumes/kubernetes.io~projected/kube-api-access-mljlb DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d0f9e9079fc53a703379391f34ab9b1ebc69d7e349be659863c7e674037a4f3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/735d44d3-7a63-4a36-8278-452074cec97c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1e67e7ae3b7a66c MacAddress:2a:b6:27:1f:b8:64 Speed:10000 Mtu:8901} {Name:249c027c5adfb01 MacAddress:c2:39:b4:6a:86:91 Speed:10000 Mtu:8901} {Name:422e06340d9787f MacAddress:ae:bb:7c:fb:68:95 Speed:10000 Mtu:8901} {Name:498f067da301d7b MacAddress:ee:fb:37:84:b0:a9 Speed:10000 Mtu:8901} {Name:5b9b500794c02c5 MacAddress:52:e7:36:1b:67:14 Speed:10000 Mtu:8901} {Name:703cf2933db32a4 MacAddress:be:78:30:07:c6:a1 Speed:10000 Mtu:8901} {Name:9d0f9e9079fc53a MacAddress:7e:e8:ed:ac:c1:80 Speed:10000 Mtu:8901} {Name:b334cc978c99067 MacAddress:96:d9:c0:6d:da:06 Speed:10000 Mtu:8901} {Name:b66fd7932ec7f32 MacAddress:56:89:cd:a8:4f:81 Speed:10000 Mtu:8901} {Name:bd8b34bfe5082da MacAddress:26:7f:22:6e:41:6f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d7:0d:b2:17 Speed:0 Mtu:9001} {Name:br-int MacAddress:46:a7:ab:fb:64:5c Speed:0 Mtu:8901} {Name:d03de8ef1759a01 MacAddress:5a:d2:41:06:7a:1d Speed:10000 Mtu:8901} {Name:e0d83eec7e5abe1 MacAddress:ce:52:e7:97:2a:b7 Speed:10000 Mtu:8901} {Name:e975873444920ca MacAddress:d6:cd:99:11:56:19 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d7:0d:b2:17 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:5b:67:bb:29:86 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:6a:e1:79:30:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368479    4913 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368524    4913 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368801    4913 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368822    4913 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-121.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368958    4913 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368966    4913 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368976    4913 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.368996    4913 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.369013    4913 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.369158    4913 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.369280    4913 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.369314    4913 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.369330    4913 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.369340    4913 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.369349    4913 kubelet.go:405] "Adding apiserver pod source"
Oct 31 07:29:11.369932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.369357    4913 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 07:29:11.370530 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.370237    4913 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 07:29:11.370530 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.370251    4913 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:29:11.370774 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.370755    4913 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 07:29:11.372988 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.372972    4913 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 07:29:11.372988 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.372989    4913 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:29:11.373107 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373096    4913 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 07:29:11.373374 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373360    4913 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 07:29:11.373425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373396    4913 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 07:29:11.373544 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373530    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 07:29:11.373573 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373557    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 07:29:11.373600 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373571    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 07:29:11.373600 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373583    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 07:29:11.373600 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373594    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 07:29:11.373677 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373608    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 07:29:11.373677 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373622    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 07:29:11.373677 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373634    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 07:29:11.373677 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373649    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 07:29:11.373677 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373662    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 07:29:11.373813 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373678    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 07:29:11.373813 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373697    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 07:29:11.373813 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373744    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 07:29:11.373813 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.373756    4913 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 07:29:11.377585 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.377567    4913 server.go:1263] "Started kubelet"
Oct 31 07:29:11.377689 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.377642    4913 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 07:29:11.377797 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.377744    4913 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 07:29:11.377894 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.377858    4913 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 07:29:11.378104 ip-10-0-130-121 systemd[1]: Started Kubernetes Kubelet.
Oct 31 07:29:11.378231 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.378165    4913 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 07:29:11.381701 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.380909    4913 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:11.382362 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.382341    4913 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 07:29:11.382550 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.382532    4913 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 07:29:11.382948 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.382929    4913 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.383721    4913 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.383737    4913 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.383785    4913 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.383949    4913 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:32 +0000 UTC" deadline="2028-05-16 03:07:21.30532495 +0000 UTC"
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.383987    4913 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13507h38m9.921341865s"
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.384452    4913 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.385556    4913 factory.go:55] Registering systemd factory
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.385574    4913 factory.go:223] Registration of the systemd container factory successfully
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.385775    4913 factory.go:153] Registering CRI-O factory
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.385786    4913 factory.go:223] Registration of the crio container factory successfully
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.385843    4913 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.385862    4913 factory.go:103] Registering Raw factory
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.385878    4913 manager.go:1199] Started watching for new ooms in manager
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386148    4913 manager.go:319] Starting recovery of all containers
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386505    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3432e0f3-ae43-4815-a8b1-38e7b7c0222d" volumeName="kubernetes.io/configmap/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-iptables-alerter-script" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386559    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3adfe0f6-05ab-44f9-8884-06c021c42f5a" volumeName="kubernetes.io/configmap/3adfe0f6-05ab-44f9-8884-06c021c42f5a-config-volume" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386575    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="021b18d4-0686-43c1-9faf-b1ca555d22a3" volumeName="kubernetes.io/projected/021b18d4-0686-43c1-9faf-b1ca555d22a3-kube-api-access-6vgww" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386588    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63139743-194a-4517-9eba-62a858a3b649" volumeName="kubernetes.io/projected/63139743-194a-4517-9eba-62a858a3b649-kube-api-access-d8d86" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386603    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735d44d3-7a63-4a36-8278-452074cec97c" volumeName="kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-config" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386616    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff9855f-fd75-41e0-82c4-3463f005a54d" volumeName="kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386629    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff9855f-fd75-41e0-82c4-3463f005a54d" volumeName="kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386642    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1a70f5-55db-4e36-959d-95bae0872fea" volumeName="kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-trusted-ca" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386657    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6121aa17-a37d-47de-af76-08c2aa174568" volumeName="kubernetes.io/secret/6121aa17-a37d-47de-af76-08c2aa174568-agent-certs" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386703    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735d44d3-7a63-4a36-8278-452074cec97c" volumeName="kubernetes.io/secret/735d44d3-7a63-4a36-8278-452074cec97c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386721    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9420864f-c0f7-47d3-ad36-b4b15ba44b57" volumeName="kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386756    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd4df54c-65e4-4dba-943c-17219493fc36" volumeName="kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-stats-auth" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386771    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3432e0f3-ae43-4815-a8b1-38e7b7c0222d" volumeName="kubernetes.io/projected/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-kube-api-access-rjrfw" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386785    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff9855f-fd75-41e0-82c4-3463f005a54d" volumeName="kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386807    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd4df54c-65e4-4dba-943c-17219493fc36" volumeName="kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-default-certificate" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386822    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="686827ed-5eb8-4cdc-aeca-176b015eb4f8" volumeName="kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386838    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87fbf516-95a9-4c2c-a41f-1a053ff98e6b" volumeName="kubernetes.io/projected/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-kube-api-access-hjlnk" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386852    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" volumeName="kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386864    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" volumeName="kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386877    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c64dab3d-c1ed-4828-b71f-d80b1f7fab00" volumeName="kubernetes.io/projected/c64dab3d-c1ed-4828-b71f-d80b1f7fab00-kube-api-access-m7dpq" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386891    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff383e3f-6228-49ad-bf4e-f43d3eaadf03" volumeName="kubernetes.io/projected/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-kube-api-access-vwmn5" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386905    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735d44d3-7a63-4a36-8278-452074cec97c" volumeName="kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386933    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87fbf516-95a9-4c2c-a41f-1a053ff98e6b" volumeName="kubernetes.io/secret/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-serving-cert" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386946    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b210005-d749-4ec8-ad41-bc7afaeb7011" volumeName="kubernetes.io/projected/8b210005-d749-4ec8-ad41-bc7afaeb7011-kube-api-access-4wgq9" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386959    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff9855f-fd75-41e0-82c4-3463f005a54d" volumeName="kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386972    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12778afe-1cd7-4829-b2c5-ddf2f1232ac2" volumeName="kubernetes.io/projected/12778afe-1cd7-4829-b2c5-ddf2f1232ac2-kube-api-access-5p5nt" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386985    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4acf6e61-938d-495e-92ea-c1bc343b9406" volumeName="kubernetes.io/configmap/4acf6e61-938d-495e-92ea-c1bc343b9406-serviceca" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.386998    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735d44d3-7a63-4a36-8278-452074cec97c" volumeName="kubernetes.io/projected/735d44d3-7a63-4a36-8278-452074cec97c-kube-api-access-mljlb" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387018    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" volumeName="kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387031    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff383e3f-6228-49ad-bf4e-f43d3eaadf03" volumeName="kubernetes.io/configmap/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-config" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387050    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59cf0458-de23-4a5e-884f-4a583e78ff38" volumeName="kubernetes.io/projected/59cf0458-de23-4a5e-884f-4a583e78ff38-kube-api-access-n4fk2" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387063    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59cf0458-de23-4a5e-884f-4a583e78ff38" volumeName="kubernetes.io/secret/59cf0458-de23-4a5e-884f-4a583e78ff38-serving-cert" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387075    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8e87b5-3eda-4fd4-9b70-f72fe97084ca" volumeName="kubernetes.io/projected/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-kube-api-access-m9fxd" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387088    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" volumeName="kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387105    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" volumeName="kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387117    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a329ba42-e8f3-43ed-9355-d906a577e176" volumeName="kubernetes.io/projected/a329ba42-e8f3-43ed-9355-d906a577e176-kube-api-access-5xsdc" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387130    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e684ad1d-ea28-4d75-aca0-8c12227ffada" volumeName="kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-tmp" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387143    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="021b18d4-0686-43c1-9faf-b1ca555d22a3" volumeName="kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-tmp" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387156    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="021b18d4-0686-43c1-9faf-b1ca555d22a3" volumeName="kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-tuned" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387171    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59cf0458-de23-4a5e-884f-4a583e78ff38" volumeName="kubernetes.io/configmap/59cf0458-de23-4a5e-884f-4a583e78ff38-config" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387207    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b210005-d749-4ec8-ad41-bc7afaeb7011" volumeName="kubernetes.io/configmap/8b210005-d749-4ec8-ad41-bc7afaeb7011-telemetry-config" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387222    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff9855f-fd75-41e0-82c4-3463f005a54d" volumeName="kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387235    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1a70f5-55db-4e36-959d-95bae0872fea" volumeName="kubernetes.io/secret/9a1a70f5-55db-4e36-959d-95bae0872fea-serving-cert" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387248    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0e85bb8-780b-4540-8e9e-19990f22f515" volumeName="kubernetes.io/projected/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-api-access-mlwg8" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387261    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bca13e-12d2-4673-8223-92f7755eac90" volumeName="kubernetes.io/projected/b5bca13e-12d2-4673-8223-92f7755eac90-kube-api-access-zl6bn" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387274    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0546b914-d103-41a5-9e5b-13c6e41bcd85" volumeName="kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387287    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3adfe0f6-05ab-44f9-8884-06c021c42f5a" volumeName="kubernetes.io/empty-dir/3adfe0f6-05ab-44f9-8884-06c021c42f5a-tmp-dir" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387300    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b3d13df-dc50-41c8-a7cd-051fb9934a34" volumeName="kubernetes.io/empty-dir/5b3d13df-dc50-41c8-a7cd-051fb9934a34-tmp-dir" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387314    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8e87b5-3eda-4fd4-9b70-f72fe97084ca" volumeName="kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-daemon-config" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387327    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1a70f5-55db-4e36-959d-95bae0872fea" volumeName="kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-config" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387341    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8e87b5-3eda-4fd4-9b70-f72fe97084ca" volumeName="kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cni-binary-copy" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387355    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6121aa17-a37d-47de-af76-08c2aa174568" volumeName="kubernetes.io/configmap/6121aa17-a37d-47de-af76-08c2aa174568-konnectivity-ca" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387368    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff9855f-fd75-41e0-82c4-3463f005a54d" volumeName="kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387380    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e684ad1d-ea28-4d75-aca0-8c12227ffada" volumeName="kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387393    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83e1b5fd-c06c-45a2-9c74-f29515e920ff" volumeName="kubernetes.io/projected/83e1b5fd-c06c-45a2-9c74-f29515e920ff-kube-api-access-cg4vv" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387407    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87fbf516-95a9-4c2c-a41f-1a053ff98e6b" volumeName="kubernetes.io/configmap/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-config" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387430    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0e85bb8-780b-4540-8e9e-19990f22f515" volumeName="kubernetes.io/empty-dir/b0e85bb8-780b-4540-8e9e-19990f22f515-data-volume" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387443    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b3d13df-dc50-41c8-a7cd-051fb9934a34" volumeName="kubernetes.io/projected/5b3d13df-dc50-41c8-a7cd-051fb9934a34-kube-api-access-pc9xp" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387457    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d264640-a684-42bd-8787-86d2382a7eb4" volumeName="kubernetes.io/projected/0d264640-a684-42bd-8787-86d2382a7eb4-kube-api-access-v5pdw" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387469    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1820478c-a566-4c78-b063-e8ceb9c6e221" volumeName="kubernetes.io/projected/1820478c-a566-4c78-b063-e8ceb9c6e221-kube-api-access-cqf8w" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387481    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3adfe0f6-05ab-44f9-8884-06c021c42f5a" volumeName="kubernetes.io/projected/3adfe0f6-05ab-44f9-8884-06c021c42f5a-kube-api-access-2brvk" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387494    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63139743-194a-4517-9eba-62a858a3b649" volumeName="kubernetes.io/secret/63139743-194a-4517-9eba-62a858a3b649-cert" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387507    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd4df54c-65e4-4dba-943c-17219493fc36" volumeName="kubernetes.io/projected/bd4df54c-65e4-4dba-943c-17219493fc36-kube-api-access-lcdmk" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387518    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c82b956f-d109-4a4e-9a90-f298f5dcaf11" volumeName="kubernetes.io/configmap/c82b956f-d109-4a4e-9a90-f298f5dcaf11-nginx-conf" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387531    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e684ad1d-ea28-4d75-aca0-8c12227ffada" volumeName="kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-service-ca-bundle" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387544    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0546b914-d103-41a5-9e5b-13c6e41bcd85" volumeName="kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387556    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0546b914-d103-41a5-9e5b-13c6e41bcd85" volumeName="kubernetes.io/projected/0546b914-d103-41a5-9e5b-13c6e41bcd85-kube-api-access-xc79k" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387568    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4acf6e61-938d-495e-92ea-c1bc343b9406" volumeName="kubernetes.io/projected/4acf6e61-938d-495e-92ea-c1bc343b9406-kube-api-access-r5nn6" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387581    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff9855f-fd75-41e0-82c4-3463f005a54d" volumeName="kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387594    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" volumeName="kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387607    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e684ad1d-ea28-4d75-aca0-8c12227ffada" volumeName="kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-snapshots" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387620    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e684ad1d-ea28-4d75-aca0-8c12227ffada" volumeName="kubernetes.io/projected/e684ad1d-ea28-4d75-aca0-8c12227ffada-kube-api-access-5hgs9" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387632    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e684ad1d-ea28-4d75-aca0-8c12227ffada" volumeName="kubernetes.io/secret/e684ad1d-ea28-4d75-aca0-8c12227ffada-serving-cert" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387645    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0546b914-d103-41a5-9e5b-13c6e41bcd85" volumeName="kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-binary-copy" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387659    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="385e4d20-8bd7-4933-8a7d-4359e0301d89" volumeName="kubernetes.io/projected/385e4d20-8bd7-4933-8a7d-4359e0301d89-kube-api-access-tmtlv" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387671    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735d44d3-7a63-4a36-8278-452074cec97c" volumeName="kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-env-overrides" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387684    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" volumeName="kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387696    4913 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1a70f5-55db-4e36-959d-95bae0872fea" volumeName="kubernetes.io/projected/9a1a70f5-55db-4e36-959d-95bae0872fea-kube-api-access-774lj" seLinuxMountContext=""
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387710    4913 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 07:29:11.388043 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.387718    4913 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 07:29:11.393177 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.393162    4913 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 07:29:11.397259 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.397042    4913 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:11.406555 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.406530    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/1.log"
Oct 31 07:29:11.406836 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.406823    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/1.log"
Oct 31 07:29:11.411543 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.411516    4913 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 07:29:11.413233 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.413212    4913 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 07:29:11.413233 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.413231    4913 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 07:29:11.413379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.413247    4913 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 07:29:11.413379 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:11.413276    4913 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 07:29:11.416412 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.416393    4913 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:11.422614 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.422584    4913 generic.go:356] "Generic (PLEG): container finished" podID="d51efa451cfc3022379daa87dd3ede24" containerID="7c422842b213567f5f2c98eb152b1b8c978b3d493d6ba84ae85baa90c9497071" exitCode=0
Oct 31 07:29:11.432403 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.432377    4913 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="43739b739f16c3c8f3ee47c48c40a13b227f0bdb9a1a63c2f09a364ce7a8354a" exitCode=0
Oct 31 07:29:11.432403 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.432400    4913 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="4462d7dc2a7fc5ac76a8c56f6fe9f37d8e434e1fade634c7b84cc3d8b76fac07" exitCode=0
Oct 31 07:29:11.432403 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.432408    4913 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="97df3bdf7b15687a90d7b9690cebe48b5672c8bc66f5c3778f57e5a69d876968" exitCode=0
Oct 31 07:29:11.432591 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.432414    4913 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="3224c9c218b40edab37a55b38b4068b9733e50e91b71655947b7d327ba20c78a" exitCode=0
Oct 31 07:29:11.432591 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.432420    4913 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="d584ae5caedcbfdfa72a819e9e2c2aa172e048053d37416a2a02dd078b684d93" exitCode=0
Oct 31 07:29:11.432591 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.432425    4913 generic.go:356] "Generic (PLEG): container finished" podID="0546b914-d103-41a5-9e5b-13c6e41bcd85" containerID="8eef32242a5cc44820dbec330d94f8b1b2fe7ae49a81caa6f1b149601328161e" exitCode=0
Oct 31 07:29:11.439121 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.439104    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/1.log"
Oct 31 07:29:11.439227 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.439128    4913 generic.go:356] "Generic (PLEG): container finished" podID="9a1a70f5-55db-4e36-959d-95bae0872fea" containerID="6914b5364454241da8b1f446be10e10463a4e06ef29c4c49f44dc9c8d5db401e" exitCode=255
Oct 31 07:29:11.478933 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.478868    4913 manager.go:324] Recovery completed
Oct 31 07:29:11.500581 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500554    4913 cpu_manager.go:226] "Starting" policy="none"
Oct 31 07:29:11.500581 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500574    4913 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 07:29:11.500726 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500594    4913 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 07:29:11.500772 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500738    4913 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 07:29:11.500772 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500753    4913 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 07:29:11.500834 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500777    4913 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 07:29:11.500834 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500785    4913 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 07:29:11.500834 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500793    4913 policy_none.go:50] "Start"
Oct 31 07:29:11.500834 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500814    4913 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 07:29:11.500834 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500829    4913 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 07:29:11.500975 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500964    4913 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 07:29:11.501006 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500976    4913 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 07:29:11.501006 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.500982    4913 policy_none.go:44] "Start"
Oct 31 07:29:11.505260 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.505233    4913 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 07:29:11.508850 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.508837    4913 manager.go:341] "Starting Device Plugin manager"
Oct 31 07:29:11.508925 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:11.508862    4913 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 07:29:11.508925 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.508871    4913 server.go:87] "Starting device plugin registration server"
Oct 31 07:29:11.509620 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.509605    4913 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 07:29:11.509663 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.509623    4913 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 07:29:11.509753 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.509735    4913 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 07:29:11.509871 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.509840    4913 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 07:29:11.509938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.509882    4913 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 07:29:11.509938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.509889    4913 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 07:29:11.510026 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.509954    4913 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:29:11.509869541Z","UUID":"3f532c64-9b7e-44b9-81dc-be45f7f0d440","Handler":null,"Name":"","Endpoint":""}
Oct 31 07:29:11.511457 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.511430    4913 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 07:29:11.511457 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.511459    4913 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 07:29:11.511581 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:11.511467    4913 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 07:29:11.513511 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.513479    4913 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"]
Oct 31 07:29:11.514474 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.514457    4913 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.514592 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.514502    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"d51efa451cfc3022379daa87dd3ede24","Type":"ContainerStarted","Data":"a4f12a608bedb8144980dc3e98093b064be94fc045f3de01ba518691c9d5949a"}
Oct 31 07:29:11.514592 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.514548    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"d51efa451cfc3022379daa87dd3ede24","Type":"ContainerDied","Data":"7c422842b213567f5f2c98eb152b1b8c978b3d493d6ba84ae85baa90c9497071"}
Oct 31 07:29:11.514792 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.514774    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"d51efa451cfc3022379daa87dd3ede24","Type":"ContainerStarted","Data":"b14dfce64a90aa78f790e735f4a689603802b3ee22d0964bc8702bdd43c86beb"}
Oct 31 07:29:11.514890 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.514877    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"a42df5a3903764c9a9afa19fe2c78097","Type":"ContainerStarted","Data":"faec1956febef47147e3b5468f6d9ae9f0d2c6fd64c8dd134c11a8a456b4e669"}
Oct 31 07:29:11.514945 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.514906    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal" event={"ID":"a42df5a3903764c9a9afa19fe2c78097","Type":"ContainerStarted","Data":"7fe1b3d7887719f095609272bea432bd413aef0b6ca9b72f0b3c1a8b8b2811b4"}
Oct 31 07:29:11.515224 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.515208    4913 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.528103 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.528077    4913 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:11.528207 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:11.528135    4913 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.528518 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.528501    4913 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:11.528585 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:11.528538    4913 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.594786 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.594750    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a42df5a3903764c9a9afa19fe2c78097-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"a42df5a3903764c9a9afa19fe2c78097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.594786 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.594789    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.594963 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.594813    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.609858 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.609834    4913 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:29:11.611046 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.611025    4913 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:29:11.611134 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.611057    4913 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:29:11.611134 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.611068    4913 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-121.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:29:11.611134 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.611108    4913 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.624086 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.624065    4913 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.624176 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.624124    4913 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.695767 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.695731    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.695767 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.695768    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.695969 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.695789    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a42df5a3903764c9a9afa19fe2c78097-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"a42df5a3903764c9a9afa19fe2c78097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.695969 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.695829    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a42df5a3903764c9a9afa19fe2c78097-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"a42df5a3903764c9a9afa19fe2c78097\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.695969 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.695836    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:11.695969 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:11.695838    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51efa451cfc3022379daa87dd3ede24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" (UID: \"d51efa451cfc3022379daa87dd3ede24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:12.369761 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.369716    4913 apiserver.go:52] "Watching apiserver"
Oct 31 07:29:12.428269 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.428239    4913 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 07:29:12.429048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.429020    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7h5mf","openshift-insights/insights-operator-7c4986d449-rjjcf","openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb","kube-system/kas-connection-checker-685895c7c9-22c6v","openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal","kube-system/konnectivity-agent-sbln7","openshift-multus/multus-hmj2s","openshift-image-registry/node-ca-fz4tp","openshift-ingress-canary/ingress-canary-w9f8b","openshift-ovn-kubernetes/ovnkube-node-vvnff","openshift-dns/dns-default-46cpb","kube-system/kas-connection-checker-685895c7c9-5nzmp","openshift-insights/insights-runtime-extractor-cdwzx","openshift-ingress/router-default-7f6f75f655-wgvm2","openshift-dns/node-resolver-49r2r","openshift-network-operator/iptables-alerter-6859w","openshift-network-diagnostics/network-check-target-d5t72","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh","openshift-image-registry/image-registry-7f4d8d5849-jrvx7","openshift-multus/network-metrics-daemon-svhj9","openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww","kube-system/kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal","openshift-network-console/networking-console-plugin-86c478bb88-jgczn","kube-system/kas-connection-checker-685895c7c9-ch85h","openshift-cluster-node-tuning-operator/tuned-nrtcz","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l","openshift-image-registry/image-registry-5b69fcb956-87gvh","openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg","openshift-multus/multus-additional-cni-plugins-d4l58","kube-system/global-pull-secret-syncer-l4tmf","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"]
Oct 31 07:29:12.433105 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.433079    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:12.433498 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.433478    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:12.433819 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.433784    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.434635 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.434618    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.436877 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.436852    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 07:29:12.436957 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.436876    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 07:29:12.436957 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.436896    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.437072 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437059    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 07:29:12.437221 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437206    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:12.437397 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437376    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 07:29:12.437477 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437442    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.437675 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437649    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p4qcx\""
Oct 31 07:29:12.437762 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437692    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.437762 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437693    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.437762 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437712    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9wzll\""
Oct 31 07:29:12.437762 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437695    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.437897 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.437857    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p7ztr\""
Oct 31 07:29:12.438154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438134    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 07:29:12.438248 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438213    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 07:29:12.438365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438347    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.438365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438363    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 07:29:12.438491 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438373    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 07:29:12.438491 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438375    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 07:29:12.438713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438693    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 07:29:12.438963 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438942    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.439066 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.438974    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6ctpv\""
Oct 31 07:29:12.439228 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.439202    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 07:29:12.440432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.440412    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lj5js\""
Oct 31 07:29:12.440700 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.440680    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.440856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.440728    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 07:29:12.440856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.440741    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.440856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.440745    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 07:29:12.440856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.440807    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.440998 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.440908    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.441078 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.441056    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.441584 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.441568    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.441669 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.441601    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.441809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.441789    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 07:29:12.442194 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.442162    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-48flb\""
Oct 31 07:29:12.442358 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.442339    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:12.442485 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.442466    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5tc29\""
Oct 31 07:29:12.443073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.443051    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jwmk9\""
Oct 31 07:29:12.443390 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.443370    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.443498 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.443477    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 07:29:12.443565 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.443495    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.443565 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.443516    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 07:29:12.443650 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.443573    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.443757 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.443730    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.444218 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444171    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ps9gq\""
Oct 31 07:29:12.444366 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444353    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 07:29:12.444522 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444502    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.444587 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444534    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 07:29:12.444716 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444698    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 07:29:12.444881 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444864    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.444945 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444868    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 07:29:12.444945 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444907    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.445035 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444960    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.445035 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444978    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zzbc9\""
Oct 31 07:29:12.445131 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.445089    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.445178 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.445135    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 07:29:12.445281 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.444872    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 07:29:12.445373 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.445269    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ghqlz\""
Oct 31 07:29:12.445373 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.445318    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 07:29:12.445517 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.445502    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-zwdw4\""
Oct 31 07:29:12.445688 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.445672    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.446491 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.446470    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k6p7p\""
Oct 31 07:29:12.446843 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.446822    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.446958 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.446938    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.447866 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.447845    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nq6dm\""
Oct 31 07:29:12.448002 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.447980    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2shdh\""
Oct 31 07:29:12.448002 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.447992    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 07:29:12.448140 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448011    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 07:29:12.448140 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448055    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zjbzq\""
Oct 31 07:29:12.448140 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448132    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x8xx7\""
Oct 31 07:29:12.448365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448140    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ht7pz\""
Oct 31 07:29:12.448365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448209    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 07:29:12.448365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448255    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 07:29:12.448365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448345    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.448577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448379    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 07:29:12.448577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448424    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.448577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448441    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 07:29:12.448577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448450    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.448577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448491    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xr95\""
Oct 31 07:29:12.448577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448499    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 07:29:12.448577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448538    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:29:12.448875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448594    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.448875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448620    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 07:29:12.448875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448771    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.448875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448819    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9r8qc\""
Oct 31 07:29:12.448875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.448860    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.449056 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.449036    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 07:29:12.449097 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.449078    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:29:12.449154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.449124    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6w9r4\""
Oct 31 07:29:12.449364 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.449345    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.450235 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.450214    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 07:29:12.450575 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.450561    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.450828 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.450803    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.451023 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.451011    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 07:29:12.451073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.451057    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bn5lf\""
Oct 31 07:29:12.451257 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.451245    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.451557 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.451542    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 07:29:12.451894 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.451878    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.452075 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.452057    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 07:29:12.452156 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.452140    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7sl8v\""
Oct 31 07:29:12.452272 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.452258    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 07:29:12.453035 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.453019    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.453097 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.453036    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.453097 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.453054    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 07:29:12.453402 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.453388    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gd7dx\""
Oct 31 07:29:12.453470 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.453438    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hfwqd\""
Oct 31 07:29:12.453527 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.453514    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 07:29:12.453559 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.453538    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 07:29:12.454089 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.454074    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 07:29:12.454388 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.454374    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-frjtn\""
Oct 31 07:29:12.454573 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.454560    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 07:29:12.454632 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.454624    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g74pl\""
Oct 31 07:29:12.454672 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.454633    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 07:29:12.454836 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.454823    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:12.467572 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.467553    4913 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:12.485793 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.485767    4913 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 07:29:12.486323 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.486309    4913 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:12.486381 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.486351    4913 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:29:12.499547 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499520    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-host-slash\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:29:12.499637 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499550    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-tmp\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.499637 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499566    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5nn6\" (UniqueName: \"kubernetes.io/projected/4acf6e61-938d-495e-92ea-c1bc343b9406-kube-api-access-r5nn6\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:29:12.499637 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499584    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.499728 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499681    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-tmp\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.499806 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499790    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.499843 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499818    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrfw\" (UniqueName: \"kubernetes.io/projected/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-kube-api-access-rjrfw\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:29:12.499888 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499842    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.499942 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.499924    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.500106 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500091    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.500158 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500123    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-k8s-cni-cncf-io\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.500226 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500156    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysconfig\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.500226 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500198    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.500294 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500235    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-log-socket\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.500323 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500287    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmn5\" (UniqueName: \"kubernetes.io/projected/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-kube-api-access-vwmn5\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:12.500359 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500334    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e684ad1d-ea28-4d75-aca0-8c12227ffada-serving-cert\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.500461 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500443    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:12.500577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500557    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-config\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.500644 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500585    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e684ad1d-ea28-4d75-aca0-8c12227ffada-serving-cert\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.500644 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500602    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8d86\" (UniqueName: \"kubernetes.io/projected/63139743-194a-4517-9eba-62a858a3b649-kube-api-access-d8d86\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:12.500644 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500634    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1a70f5-55db-4e36-959d-95bae0872fea-serving-cert\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:12.500644 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500638    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:12.500824 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500678    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:12.500824 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500705    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdmk\" (UniqueName: \"kubernetes.io/projected/bd4df54c-65e4-4dba-943c-17219493fc36-kube-api-access-lcdmk\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.500824 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500730    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg4vv\" (UniqueName: \"kubernetes.io/projected/83e1b5fd-c06c-45a2-9c74-f29515e920ff-kube-api-access-cg4vv\") pod \"volume-data-source-validator-58c9f5f46d-t6dgh\" (UID: \"83e1b5fd-c06c-45a2-9c74-f29515e920ff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"
Oct 31 07:29:12.500824 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500753    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.500824 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500811    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-service-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500838    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cnibin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500865    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-registration-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500892    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-env-overrides\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500903    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1a70f5-55db-4e36-959d-95bae0872fea-serving-cert\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500919    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fk2\" (UniqueName: \"kubernetes.io/projected/59cf0458-de23-4a5e-884f-4a583e78ff38-kube-api-access-n4fk2\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500925    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500898    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-config\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500967    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-config\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:12.501058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.500995    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501127    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501242    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501266    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-env-overrides\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501270    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/735d44d3-7a63-4a36-8278-452074cec97c-ovn-node-metrics-cert\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501289    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-config\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501347    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqf8w\" (UniqueName: \"kubernetes.io/projected/1820478c-a566-4c78-b063-e8ceb9c6e221-kube-api-access-cqf8w\") pod \"kas-connection-checker-685895c7c9-5nzmp\" (UID: \"1820478c-a566-4c78-b063-e8ceb9c6e221\") " pod="kube-system/kas-connection-checker-685895c7c9-5nzmp"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501397    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7bd\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501431    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-service-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501431    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-device-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.501500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501493    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501615    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501635    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brvk\" (UniqueName: \"kubernetes.io/projected/3adfe0f6-05ab-44f9-8884-06c021c42f5a-kube-api-access-2brvk\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501650    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9xm\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501711    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501723    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/735d44d3-7a63-4a36-8278-452074cec97c-ovn-node-metrics-cert\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501734    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501792    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc79k\" (UniqueName: \"kubernetes.io/projected/0546b914-d103-41a5-9e5b-13c6e41bcd85-kube-api-access-xc79k\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501821    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501861    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fxd\" (UniqueName: \"kubernetes.io/projected/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-kube-api-access-m9fxd\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501904    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgww\" (UniqueName: \"kubernetes.io/projected/021b18d4-0686-43c1-9faf-b1ca555d22a3-kube-api-access-6vgww\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501951    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwg8\" (UniqueName: \"kubernetes.io/projected/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-api-access-mlwg8\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.501999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.501982    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-trusted-ca\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502008    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6121aa17-a37d-47de-af76-08c2aa174568-agent-certs\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502035    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502060    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-cnibin\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502087    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-daemon-config\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502116    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-var-lib-kubelet\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502138    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502155    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-script-lib\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502170    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-snapshots\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502243    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-kubelet-config\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502298    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cni-binary-copy\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502316    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1a70f5-55db-4e36-959d-95bae0872fea-trusted-ca\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502361    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-conf-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502376    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-snapshots\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502393    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6121aa17-a37d-47de-af76-08c2aa174568-agent-certs\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502405    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9xp\" (UniqueName: \"kubernetes.io/projected/5b3d13df-dc50-41c8-a7cd-051fb9934a34-kube-api-access-pc9xp\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502424    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-daemon-config\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502430    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-system-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502477    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.502515 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502504    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502528    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cni-binary-copy\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502536    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-multus-certs\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502573    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59cf0458-de23-4a5e-884f-4a583e78ff38-serving-cert\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502616    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4acf6e61-938d-495e-92ea-c1bc343b9406-host\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502644    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/735d44d3-7a63-4a36-8278-452074cec97c-ovnkube-script-lib\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502706    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-stats-auth\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502732    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-tuned\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502748    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-slash\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502811    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-node-log\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502829    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5nt\" (UniqueName: \"kubernetes.io/projected/12778afe-1cd7-4829-b2c5-ddf2f1232ac2-kube-api-access-5p5nt\") pod \"migrator-66df5d496b-tbdww\" (UID: \"12778afe-1cd7-4829-b2c5-ddf2f1232ac2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502844    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502860    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502888    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59cf0458-de23-4a5e-884f-4a583e78ff38-serving-cert\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502887    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6bn\" (UniqueName: \"kubernetes.io/projected/b5bca13e-12d2-4673-8223-92f7755eac90-kube-api-access-zl6bn\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502915    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:12.503145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502940    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-tuned\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.503809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.502941    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:12.503809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503310    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-sys-fs\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.503809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503375    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b3d13df-dc50-41c8-a7cd-051fb9934a34-hosts-file\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:29:12.503809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503410    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.503809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503426    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.503809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503438    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtlv\" (UniqueName: \"kubernetes.io/projected/385e4d20-8bd7-4933-8a7d-4359e0301d89-kube-api-access-tmtlv\") pod \"kas-connection-checker-685895c7c9-22c6v\" (UID: \"385e4d20-8bd7-4933-8a7d-4359e0301d89\") " pod="kube-system/kas-connection-checker-685895c7c9-22c6v"
Oct 31 07:29:12.503809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503462    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-bin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.504150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503816    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.504150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503875    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.504150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503932    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.504150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503957    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-stats-auth\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.504150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.503973    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-bin\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.504150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504025    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-netd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.504434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504161    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.504434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504217    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.504434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504223    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3d13df-dc50-41c8-a7cd-051fb9934a34-tmp-dir\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:29:12.504434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504315    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-kubernetes\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.504434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504331    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b3d13df-dc50-41c8-a7cd-051fb9934a34-tmp-dir\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:29:12.504434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504362    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.504434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504375    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3adfe0f6-05ab-44f9-8884-06c021c42f5a-tmp-dir\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.504782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504538    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3adfe0f6-05ab-44f9-8884-06c021c42f5a-tmp-dir\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.504782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504608    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-kubelet\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.504782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504691    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6121aa17-a37d-47de-af76-08c2aa174568-konnectivity-ca\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:29:12.504782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504707    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.504782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504721    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:12.505000 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504914    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.505000 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504918    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6121aa17-a37d-47de-af76-08c2aa174568-konnectivity-ca\") pod \"konnectivity-agent-sbln7\" (UID: \"6121aa17-a37d-47de-af76-08c2aa174568\") " pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:29:12.505000 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504966    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-conf\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.505127 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.504996    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-lib-modules\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.505127 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505056    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774lj\" (UniqueName: \"kubernetes.io/projected/9a1a70f5-55db-4e36-959d-95bae0872fea-kube-api-access-774lj\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:12.505127 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505104    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.505273 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505130    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:12.505273 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505204    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.505273 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505237    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgs9\" (UniqueName: \"kubernetes.io/projected/e684ad1d-ea28-4d75-aca0-8c12227ffada-kube-api-access-5hgs9\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.505273 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505267    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-multus\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.505462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505298    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-sys\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.505462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505331    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e85bb8-780b-4540-8e9e-19990f22f515-data-volume\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.505462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505452    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-systemd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.505599 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505487    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4acf6e61-938d-495e-92ea-c1bc343b9406-serviceca\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:29:12.505599 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505527    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-socket-dir-parent\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.505599 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505565    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-run\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.505781 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505604    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59cf0458-de23-4a5e-884f-4a583e78ff38-config\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:12.505781 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505639    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-systemd-units\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.505781 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505691    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljlb\" (UniqueName: \"kubernetes.io/projected/735d44d3-7a63-4a36-8278-452074cec97c-kube-api-access-mljlb\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.505781 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505719    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.505781 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505749    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsdc\" (UniqueName: \"kubernetes.io/projected/a329ba42-e8f3-43ed-9355-d906a577e176-kube-api-access-5xsdc\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.505984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505780    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.505984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505809    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-netns\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.505984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505837    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-kubelet\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.505984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505877    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-netns\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.505984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505906    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63139743-194a-4517-9eba-62a858a3b649-cert\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:12.505984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505966    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-systemd\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.506255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.505996    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-system-cni-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.506255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.506046    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b210005-d749-4ec8-ad41-bc7afaeb7011-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:12.506255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.506073    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:12.506255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.506101    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-dbus\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:12.506255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.506132    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3adfe0f6-05ab-44f9-8884-06c021c42f5a-config-volume\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.506255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.506177    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.506525 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.506307    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.506719 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.506698    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63139743-194a-4517-9eba-62a858a3b649-cert\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:12.506961 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.506942    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.507151 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.507100    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9420864f-c0f7-47d3-ad36-b4b15ba44b57-original-pull-secret\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:12.507270 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.507247    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrfw\" (UniqueName: \"kubernetes.io/projected/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-kube-api-access-rjrfw\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:29:12.507496 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.507477    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3adfe0f6-05ab-44f9-8884-06c021c42f5a-config-volume\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.507578 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.507557    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b210005-d749-4ec8-ad41-bc7afaeb7011-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:12.507721 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.507702    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8d86\" (UniqueName: \"kubernetes.io/projected/63139743-194a-4517-9eba-62a858a3b649-kube-api-access-d8d86\") pod \"ingress-canary-w9f8b\" (UID: \"63139743-194a-4517-9eba-62a858a3b649\") " pod="openshift-ingress-canary/ingress-canary-w9f8b"
Oct 31 07:29:12.508004 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.507982    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5nn6\" (UniqueName: \"kubernetes.io/projected/4acf6e61-938d-495e-92ea-c1bc343b9406-kube-api-access-r5nn6\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:29:12.508245 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508224    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4acf6e61-938d-495e-92ea-c1bc343b9406-serviceca\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:29:12.508354 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508335    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmn5\" (UniqueName: \"kubernetes.io/projected/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-kube-api-access-vwmn5\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:12.508501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508485    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59cf0458-de23-4a5e-884f-4a583e78ff38-config\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:12.508561 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508520    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0546b914-d103-41a5-9e5b-13c6e41bcd85-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.508667 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508643    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-tmp\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.508733 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508691    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-iptables-alerter-script\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:29:12.508733 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508726    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-modprobe-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.508856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508825    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e684ad1d-ea28-4d75-aca0-8c12227ffada-tmp\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.508907 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508862    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e85bb8-780b-4540-8e9e-19990f22f515-data-volume\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.508907 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508895    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.509061 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.508940    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-default-certificate\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.509256 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509238    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.509319 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509269    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-iptables-alerter-script\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:29:12.509368 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509329    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgq9\" (UniqueName: \"kubernetes.io/projected/8b210005-d749-4ec8-ad41-bc7afaeb7011-kube-api-access-4wgq9\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:12.509436 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509386    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.509486 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509424    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-host\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.509536 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509491    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fk2\" (UniqueName: \"kubernetes.io/projected/59cf0458-de23-4a5e-884f-4a583e78ff38-kube-api-access-n4fk2\") pod \"service-ca-operator-544d749c8c-ttcjb\" (UID: \"59cf0458-de23-4a5e-884f-4a583e78ff38\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb"
Oct 31 07:29:12.509536 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509484    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7dpq\" (UniqueName: \"kubernetes.io/projected/c64dab3d-c1ed-4828-b71f-d80b1f7fab00-kube-api-access-m7dpq\") pod \"network-check-source-6f78d59f99-9sbwg\" (UID: \"c64dab3d-c1ed-4828-b71f-d80b1f7fab00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"
Oct 31 07:29:12.509657 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509574    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pdw\" (UniqueName: \"kubernetes.io/projected/0d264640-a684-42bd-8787-86d2382a7eb4-kube-api-access-v5pdw\") pod \"kas-connection-checker-685895c7c9-ch85h\" (UID: \"0d264640-a684-42bd-8787-86d2382a7eb4\") " pod="kube-system/kas-connection-checker-685895c7c9-ch85h"
Oct 31 07:29:12.509657 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509610    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-hostroot\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.509657 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509636    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0e85bb8-780b-4540-8e9e-19990f22f515-crio-socket\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.509812 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509673    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.509812 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509710    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-etc-kubernetes\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.509812 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509675    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg4vv\" (UniqueName: \"kubernetes.io/projected/83e1b5fd-c06c-45a2-9c74-f29515e920ff-kube-api-access-cg4vv\") pod \"volume-data-source-validator-58c9f5f46d-t6dgh\" (UID: \"83e1b5fd-c06c-45a2-9c74-f29515e920ff\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-t6dgh"
Oct 31 07:29:12.509812 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509745    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:12.509812 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509775    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-etc-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.510048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509805    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.510048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509848    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-default-certificate\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.510048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509848    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlnk\" (UniqueName: \"kubernetes.io/projected/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-kube-api-access-hjlnk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:12.510048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509940    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c82b956f-d109-4a4e-9a90-f298f5dcaf11-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:12.510048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.509974    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-var-lib-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.510048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.510008    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-ovn\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.510339 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.510058    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-socket-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.510339 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.510156    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e684ad1d-ea28-4d75-aca0-8c12227ffada-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.510617 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.510591    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdmk\" (UniqueName: \"kubernetes.io/projected/bd4df54c-65e4-4dba-943c-17219493fc36-kube-api-access-lcdmk\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.510617 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.510584    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.510725 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.510631    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c82b956f-d109-4a4e-9a90-f298f5dcaf11-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:12.510725 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.510673    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:12.511496 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.511470    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqf8w\" (UniqueName: \"kubernetes.io/projected/1820478c-a566-4c78-b063-e8ceb9c6e221-kube-api-access-cqf8w\") pod \"kas-connection-checker-685895c7c9-5nzmp\" (UID: \"1820478c-a566-4c78-b063-e8ceb9c6e221\") " pod="kube-system/kas-connection-checker-685895c7c9-5nzmp"
Oct 31 07:29:12.511583 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.511525    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgww\" (UniqueName: \"kubernetes.io/projected/021b18d4-0686-43c1-9faf-b1ca555d22a3-kube-api-access-6vgww\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.511862 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.511841    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7bd\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.513776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.513294    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6bn\" (UniqueName: \"kubernetes.io/projected/b5bca13e-12d2-4673-8223-92f7755eac90-kube-api-access-zl6bn\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:12.513776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.513316    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fxd\" (UniqueName: \"kubernetes.io/projected/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-kube-api-access-m9fxd\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.513776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.513314    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brvk\" (UniqueName: \"kubernetes.io/projected/3adfe0f6-05ab-44f9-8884-06c021c42f5a-kube-api-access-2brvk\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.513776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.513486    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9xm\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.513776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.513512    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc79k\" (UniqueName: \"kubernetes.io/projected/0546b914-d103-41a5-9e5b-13c6e41bcd85-kube-api-access-xc79k\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.513776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.513684    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j988n\" (UniqueName: \"kubernetes.io/projected/686827ed-5eb8-4cdc-aeca-176b015eb4f8-kube-api-access-j988n\") pod \"network-check-target-d5t72\" (UID: \"686827ed-5eb8-4cdc-aeca-176b015eb4f8\") " pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:12.513776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.513752    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwg8\" (UniqueName: \"kubernetes.io/projected/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-api-access-mlwg8\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.513776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.513756    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5nt\" (UniqueName: \"kubernetes.io/projected/12778afe-1cd7-4829-b2c5-ddf2f1232ac2-kube-api-access-5p5nt\") pod \"migrator-66df5d496b-tbdww\" (UID: \"12778afe-1cd7-4829-b2c5-ddf2f1232ac2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tbdww"
Oct 31 07:29:12.514127 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.514074    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9xp\" (UniqueName: \"kubernetes.io/projected/5b3d13df-dc50-41c8-a7cd-051fb9934a34-kube-api-access-pc9xp\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:29:12.514548 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.514529    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.514624 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.514562    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtlv\" (UniqueName: \"kubernetes.io/projected/385e4d20-8bd7-4933-8a7d-4359e0301d89-kube-api-access-tmtlv\") pod \"kas-connection-checker-685895c7c9-22c6v\" (UID: \"385e4d20-8bd7-4933-8a7d-4359e0301d89\") " pod="kube-system/kas-connection-checker-685895c7c9-22c6v"
Oct 31 07:29:12.514835 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.514810    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsdc\" (UniqueName: \"kubernetes.io/projected/a329ba42-e8f3-43ed-9355-d906a577e176-kube-api-access-5xsdc\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.515271 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.515253    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljlb\" (UniqueName: \"kubernetes.io/projected/735d44d3-7a63-4a36-8278-452074cec97c-kube-api-access-mljlb\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.515755 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.515737    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.515913 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.515895    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774lj\" (UniqueName: \"kubernetes.io/projected/9a1a70f5-55db-4e36-959d-95bae0872fea-kube-api-access-774lj\") pod \"console-operator-7cd56b8dbc-7h5mf\" (UID: \"9a1a70f5-55db-4e36-959d-95bae0872fea\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:12.516249 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.516226    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgs9\" (UniqueName: \"kubernetes.io/projected/e684ad1d-ea28-4d75-aca0-8c12227ffada-kube-api-access-5hgs9\") pod \"insights-operator-7c4986d449-rjjcf\" (UID: \"e684ad1d-ea28-4d75-aca0-8c12227ffada\") " pod="openshift-insights/insights-operator-7c4986d449-rjjcf"
Oct 31 07:29:12.516846 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.516829    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlnk\" (UniqueName: \"kubernetes.io/projected/87fbf516-95a9-4c2c-a41f-1a053ff98e6b-kube-api-access-hjlnk\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-l546l\" (UID: \"87fbf516-95a9-4c2c-a41f-1a053ff98e6b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l"
Oct 31 07:29:12.517122 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.517104    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7dpq\" (UniqueName: \"kubernetes.io/projected/c64dab3d-c1ed-4828-b71f-d80b1f7fab00-kube-api-access-m7dpq\") pod \"network-check-source-6f78d59f99-9sbwg\" (UID: \"c64dab3d-c1ed-4828-b71f-d80b1f7fab00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9sbwg"
Oct 31 07:29:12.517529 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.517511    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pdw\" (UniqueName: \"kubernetes.io/projected/0d264640-a684-42bd-8787-86d2382a7eb4-kube-api-access-v5pdw\") pod \"kas-connection-checker-685895c7c9-ch85h\" (UID: \"0d264640-a684-42bd-8787-86d2382a7eb4\") " pod="kube-system/kas-connection-checker-685895c7c9-ch85h"
Oct 31 07:29:12.517584 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.517536    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgq9\" (UniqueName: \"kubernetes.io/projected/8b210005-d749-4ec8-ad41-bc7afaeb7011-kube-api-access-4wgq9\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:12.610665 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610632    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.610665 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610669    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610693    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-sys-fs\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610715    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b3d13df-dc50-41c8-a7cd-051fb9934a34-hosts-file\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610742    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-bin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610766    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610770    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-sys-fs\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610783    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610791    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610784    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b3d13df-dc50-41c8-a7cd-051fb9934a34-hosts-file\") pod \"node-resolver-49r2r\" (UID: \"5b3d13df-dc50-41c8-a7cd-051fb9934a34\") " pod="openshift-dns/node-resolver-49r2r"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610825    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-bin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610833    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610832    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-bin\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610866    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-bin\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.610874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610874    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610892    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-netd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610915    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-kubernetes\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.610833    4913 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610947    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-cni-netd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610956    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-kubelet\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610990    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-kubernetes\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.610931    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-kubelet\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611020    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert podName:c82b956f-d109-4a4e-9a90-f298f5dcaf11 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.11099537 +0000 UTC m=+1.800763658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jgczn" (UID: "c82b956f-d109-4a4e-9a90-f298f5dcaf11") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611061    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611103    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611129    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-conf\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611153    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-lib-modules\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611156    4913 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611195    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611223    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-multus\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611223    4913 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611261    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls podName:8b210005-d749-4ec8-ad41-bc7afaeb7011 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.111240983 +0000 UTC m=+1.801009268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qrsxv" (UID: "8b210005-d749-4ec8-ad41-bc7afaeb7011") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611272    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-cni-multus\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611289    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-lib-modules\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611281    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-sys\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611319    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.111308882 +0000 UTC m=+1.801077158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611321    4913 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611331    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-sys\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611345    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-systemd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611370    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.111354732 +0000 UTC m=+1.801123017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611383    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-conf\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611387    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-systemd\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611385    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-socket-dir-parent\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611419    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-run\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611416    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-socket-dir-parent\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.611432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611439    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-systemd-units\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611456    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-netns\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611472    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-kubelet\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611486    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-netns\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611489    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-run\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611504    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-systemd\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611516    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-systemd-units\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611509    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-netns\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611527    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-system-cni-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611538    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-var-lib-kubelet\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611555    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-system-cni-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611535    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-run-netns\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611567    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-dbus\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611574    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-systemd\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611595    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611608    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-dbus\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611622    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-modprobe-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611649    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-host\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611663    4913 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611674    4913 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611673    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-hostroot\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611699    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0e85bb8-780b-4540-8e9e-19990f22f515-crio-socket\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611715    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-host\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611717    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-modprobe-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.611723    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.111709639 +0000 UTC m=+1.801477936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611734    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-hostroot\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611747    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-etc-kubernetes\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611752    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b0e85bb8-780b-4540-8e9e-19990f22f515-crio-socket\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611776    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-etc-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611809    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-etc-kubernetes\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611803    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-etc-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611842    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611868    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-var-lib-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611883    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-ovn\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611905    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-socket-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611923    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-host-slash\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611942    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-var-lib-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611956    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-ovn\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611965    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611980    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-run-openvswitch\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611986    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612034    4913 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612034    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611991    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-socket-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612046    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.112032024 +0000 UTC m=+1.801800322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.611964    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3432e0f3-ae43-4815-a8b1-38e7b7c0222d-host-slash\") pod \"iptables-alerter-6859w\" (UID: \"3432e0f3-ae43-4815-a8b1-38e7b7c0222d\") " pod="openshift-network-operator/iptables-alerter-6859w"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612082    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.112072401 +0000 UTC m=+1.801840679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : secret "router-metrics-certs-default" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612099    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612106    4913 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612126    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-k8s-cni-cncf-io\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612145    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.112134009 +0000 UTC m=+1.801902293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612142    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612162    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-k8s-cni-cncf-io\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612203    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysconfig\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612221    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612237    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-log-socket\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612259    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cnibin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612299    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysconfig\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612298    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-log-socket\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612298    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-registration-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612317    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-etc-sysctl-d\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612330    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-cnibin\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612337    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612353    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-registration-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612371    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-device-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612388    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612405    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612421    4913 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612428    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-cnibin\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612434    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a329ba42-e8f3-43ed-9355-d906a577e176-device-dir\") pod \"aws-ebs-csi-driver-node-mqs7w\" (UID: \"a329ba42-e8f3-43ed-9355-d906a577e176\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqs7w"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612434    4913 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612462    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-cnibin\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612472    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.112462851 +0000 UTC m=+1.802231121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612468    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-var-lib-kubelet\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612485    4913 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612499    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612513    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021b18d4-0686-43c1-9faf-b1ca555d22a3-var-lib-kubelet\") pod \"tuned-nrtcz\" (UID: \"021b18d4-0686-43c1-9faf-b1ca555d22a3\") " pod="openshift-cluster-node-tuning-operator/tuned-nrtcz"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612475    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0546b914-d103-41a5-9e5b-13c6e41bcd85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4l58\" (UID: \"0546b914-d103-41a5-9e5b-13c6e41bcd85\") " pod="openshift-multus/multus-additional-cni-plugins-d4l58"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:12.612531    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.11251693 +0000 UTC m=+1.802285225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : secret "metrics-daemon-secret" not found
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612548    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-kubelet-config\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612573    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-conf-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.612542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612596    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9420864f-c0f7-47d3-ad36-b4b15ba44b57-kubelet-config\") pod \"global-pull-secret-syncer-l4tmf\" (UID: \"9420864f-c0f7-47d3-ad36-b4b15ba44b57\") " pod="kube-system/global-pull-secret-syncer-l4tmf"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612637    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-conf-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612639    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-system-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612601    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-system-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612682    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612709    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-multus-certs\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612736    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-host-run-multus-certs\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612739    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4acf6e61-938d-495e-92ea-c1bc343b9406-host\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612753    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b8e87b5-3eda-4fd4-9b70-f72fe97084ca-multus-cni-dir\") pod \"multus-hmj2s\" (UID: \"5b8e87b5-3eda-4fd4-9b70-f72fe97084ca\") " pod="openshift-multus/multus-hmj2s"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612764    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4acf6e61-938d-495e-92ea-c1bc343b9406-host\") pod \"node-ca-fz4tp\" (UID: \"4acf6e61-938d-495e-92ea-c1bc343b9406\") " pod="openshift-image-registry/node-ca-fz4tp"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612767    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-slash\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612790    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-node-log\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612794    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-host-slash\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612823    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/735d44d3-7a63-4a36-8278-452074cec97c-node-log\") pod \"ovnkube-node-vvnff\" (UID: \"735d44d3-7a63-4a36-8278-452074cec97c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:12.614789 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.612961    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b0e85bb8-780b-4540-8e9e-19990f22f515-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:12.734760 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:12.734727    4913 scope.go:122] "RemoveContainer" containerID="6914b5364454241da8b1f446be10e10463a4e06ef29c4c49f44dc9c8d5db401e"
Oct 31 07:29:13.118099 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118015    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:13.118099 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118056    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:13.118099 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118077    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118121    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118152    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118180    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.118162938 +0000 UTC m=+2.807931232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118233    4913 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118207    4913 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118267    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118275    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.118263138 +0000 UTC m=+2.808031407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : secret "metrics-daemon-secret" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118274    4913 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118296    4913 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118299    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.11828799 +0000 UTC m=+2.808056264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : secret "router-metrics-certs-default" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118331    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.118317698 +0000 UTC m=+2.808085985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118336    4913 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118375    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.118364208 +0000 UTC m=+2.808132501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:13.118386 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118374    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118401    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118420    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118494    4913 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118499    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118506    4913 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118523    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.118516405 +0000 UTC m=+2.808284675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118520    4913 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118537    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.118531913 +0000 UTC m=+2.808300183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118564    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls podName:8b210005-d749-4ec8-ad41-bc7afaeb7011 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.118551961 +0000 UTC m=+2.808320248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qrsxv" (UID: "8b210005-d749-4ec8-ad41-bc7afaeb7011") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118568    4913 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118578    4913 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.118602    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.118594657 +0000 UTC m=+2.808362931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:13.118944 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.118601    4913 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 07:29:13.120770 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.120750    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c82b956f-d109-4a4e-9a90-f298f5dcaf11-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jgczn\" (UID: \"c82b956f-d109-4a4e-9a90-f298f5dcaf11\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:13.337992 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.337955    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn"
Oct 31 07:29:13.472105 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.472075    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-49r2r_5b3d13df-dc50-41c8-a7cd-051fb9934a34/dns-node-resolver/0.log"
Oct 31 07:29:13.472604 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.472584    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:29:13.473011 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.472994    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/1.log"
Oct 31 07:29:13.473061 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.473029    4913 generic.go:356] "Generic (PLEG): container finished" podID="9a1a70f5-55db-4e36-959d-95bae0872fea" containerID="1104fe2e8bf21d08c9e097fd7b74a1b449cdf51197e991d3ec9f97eda867152b" exitCode=255
Oct 31 07:29:13.473123 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.473103    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" event={"ID":"9a1a70f5-55db-4e36-959d-95bae0872fea","Type":"ContainerDied","Data":"1104fe2e8bf21d08c9e097fd7b74a1b449cdf51197e991d3ec9f97eda867152b"}
Oct 31 07:29:13.473170 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.473160    4913 scope.go:122] "RemoveContainer" containerID="6914b5364454241da8b1f446be10e10463a4e06ef29c4c49f44dc9c8d5db401e"
Oct 31 07:29:13.473562 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.473541    4913 scope.go:122] "RemoveContainer" containerID="1104fe2e8bf21d08c9e097fd7b74a1b449cdf51197e991d3ec9f97eda867152b"
Oct 31 07:29:13.473775 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:13.473754    4913 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7h5mf_openshift-console-operator(9a1a70f5-55db-4e36-959d-95bae0872fea)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" podUID="9a1a70f5-55db-4e36-959d-95bae0872fea"
Oct 31 07:29:13.624388 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.624305    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jgczn"]
Oct 31 07:29:13.628095 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:13.628065    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82b956f_d109_4a4e_9a90_f298f5dcaf11.slice/crio-32c60ca039f0bc6dc26410ce2ad27112d8a1328e0c29b724feb899b401daa5ae WatchSource:0}: Error finding container 32c60ca039f0bc6dc26410ce2ad27112d8a1328e0c29b724feb899b401daa5ae: Status 404 returned error can't find the container with id 32c60ca039f0bc6dc26410ce2ad27112d8a1328e0c29b724feb899b401daa5ae
Oct 31 07:29:13.630129 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.630113    4913 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:29:13.678924 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:13.678901    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:14.128795 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.128742    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:14.128795 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.128797    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.128827    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.128872    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.128905    4913 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.128920    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.128943    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.128949    4913 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.128969    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.128984    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls podName:8b210005-d749-4ec8-ad41-bc7afaeb7011 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.128963436 +0000 UTC m=+4.818731724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-qrsxv" (UID: "8b210005-d749-4ec8-ad41-bc7afaeb7011") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129025    4913 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129036    4913 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129060    4913 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129101    4913 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129140    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.129128113 +0000 UTC m=+4.818896399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129158    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.129150099 +0000 UTC m=+4.818918368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:14.129163 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129174    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.12916609 +0000 UTC m=+4.818934360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129217    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.129208993 +0000 UTC m=+4.818977263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129234    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.129227196 +0000 UTC m=+4.818995474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.129282    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.129315    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129390    4913 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129402    4913 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129446    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.129433738 +0000 UTC m=+4.819202026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129449    4913 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:14.129811 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:14.129482    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs podName:b5bca13e-12d2-4673-8223-92f7755eac90 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:16.129472861 +0000 UTC m=+4.819241147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs") pod "network-metrics-daemon-svhj9" (UID: "b5bca13e-12d2-4673-8223-92f7755eac90") : secret "metrics-daemon-secret" not found
Oct 31 07:29:14.131794 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.131771    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4df54c-65e4-4dba-943c-17219493fc36-metrics-certs\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:14.473288 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.473249    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fz4tp_4acf6e61-938d-495e-92ea-c1bc343b9406/node-ca/0.log"
Oct 31 07:29:14.478113 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.478079    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:29:14.480142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:14.480105    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn" event={"ID":"c82b956f-d109-4a4e-9a90-f298f5dcaf11","Type":"ContainerStarted","Data":"32c60ca039f0bc6dc26410ce2ad27112d8a1328e0c29b724feb899b401daa5ae"}
Oct 31 07:29:15.343276 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:15.343236    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:15.473369 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:15.473343    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w9f8b_63139743-194a-4517-9eba-62a858a3b649/serve-healthcheck-canary/0.log"
Oct 31 07:29:15.485044 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:15.485008    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jgczn" event={"ID":"c82b956f-d109-4a4e-9a90-f298f5dcaf11","Type":"ContainerStarted","Data":"39966ccdfc57ba3d87160b8c869d5505ea6f2261f8ae287933b4cad4e823f357"}
Oct 31 07:29:15.953610 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:15.953578    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:15.953911 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:15.953895    4913 scope.go:122] "RemoveContainer" containerID="1104fe2e8bf21d08c9e097fd7b74a1b449cdf51197e991d3ec9f97eda867152b"
Oct 31 07:29:15.954740 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:15.954719    4913 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7h5mf_openshift-console-operator(9a1a70f5-55db-4e36-959d-95bae0872fea)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" podUID="9a1a70f5-55db-4e36-959d-95bae0872fea"
Oct 31 07:29:16.072701 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.072676    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-tbdww_12778afe-1cd7-4829-b2c5-ddf2f1232ac2/migrator/0.log"
Oct 31 07:29:16.148796 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.148758    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:16.148928 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.148806    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:16.148928 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.148826    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:16.148928 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.148850    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:16.148928 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.148867    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:16.148928 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.148905    4913 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:16.148928 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.148917    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:16.148928 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.148924    4913 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b69fcb956-87gvh: secret "image-registry-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.148929    4913 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.148950    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.148930103 +0000 UTC m=+8.838698386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.148975    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.149010    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.148978    4913 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.149045    4913 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.149059    4913 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d8d5849-jrvx7: secret "image-registry-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.149073    4913 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.149051    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls podName:92dd05ff-369c-40dc-9afd-6b54ddbc4f62 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.149026557 +0000 UTC m=+8.838794845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls") pod "image-registry-5b69fcb956-87gvh" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62") : secret "image-registry-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.149128    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls podName:8ff9855f-fd75-41e0-82c4-3463f005a54d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.149110861 +0000 UTC m=+8.838879134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls") pod "image-registry-7f4d8d5849-jrvx7" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d") : secret "image-registry-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.149147    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls podName:ff383e3f-6228-49ad-bf4e-f43d3eaadf03 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.149138276 +0000 UTC m=+8.838906553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mhwtk" (UID: "ff383e3f-6228-49ad-bf4e-f43d3eaadf03") : secret "samples-operator-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.149166    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls podName:3adfe0f6-05ab-44f9-8884-06c021c42f5a nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.1491569 +0000 UTC m=+8.838925170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls") pod "dns-default-46cpb" (UID: "3adfe0f6-05ab-44f9-8884-06c021c42f5a") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:16.149311 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:16.149210    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.149180501 +0000 UTC m=+8.838948770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:16.151301 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.151281    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5bca13e-12d2-4673-8223-92f7755eac90-metrics-certs\") pod \"network-metrics-daemon-svhj9\" (UID: \"b5bca13e-12d2-4673-8223-92f7755eac90\") " pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:16.151387 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.151281    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b210005-d749-4ec8-ad41-bc7afaeb7011-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-qrsxv\" (UID: \"8b210005-d749-4ec8-ad41-bc7afaeb7011\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:16.273750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.273674    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-l546l_87fbf516-95a9-4c2c-a41f-1a053ff98e6b/kube-storage-version-migrator-operator/0.log"
Oct 31 07:29:16.333735 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.333702    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"
Oct 31 07:29:16.342864 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.342843    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svhj9"
Oct 31 07:29:16.604055 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.603990    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv"]
Oct 31 07:29:16.625112 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:16.625076    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b210005_d749_4ec8_ad41_bc7afaeb7011.slice/crio-ecae46fdce466451303b6a27e76d5b8666a65e6172497683a2694c1fc32552b0 WatchSource:0}: Error finding container ecae46fdce466451303b6a27e76d5b8666a65e6172497683a2694c1fc32552b0: Status 404 returned error can't find the container with id ecae46fdce466451303b6a27e76d5b8666a65e6172497683a2694c1fc32552b0
Oct 31 07:29:16.625797 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.625772    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svhj9"]
Oct 31 07:29:16.628672 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:16.628615    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bca13e_12d2_4673_8223_92f7755eac90.slice/crio-18ef4ff4e8ba932dac6ca25db1f95b3e692150d6aff1152b94795289f2dee692 WatchSource:0}: Error finding container 18ef4ff4e8ba932dac6ca25db1f95b3e692150d6aff1152b94795289f2dee692: Status 404 returned error can't find the container with id 18ef4ff4e8ba932dac6ca25db1f95b3e692150d6aff1152b94795289f2dee692
Oct 31 07:29:16.800908 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.800873    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:16.801141 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:16.801125    4913 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:17.332992 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:17.332950    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvnff"
Oct 31 07:29:17.493740 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:17.493699    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv" event={"ID":"8b210005-d749-4ec8-ad41-bc7afaeb7011","Type":"ContainerStarted","Data":"ecae46fdce466451303b6a27e76d5b8666a65e6172497683a2694c1fc32552b0"}
Oct 31 07:29:17.495996 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:17.495956    4913 generic.go:356] "Generic (PLEG): container finished" podID="59cf0458-de23-4a5e-884f-4a583e78ff38" containerID="bc9f81c3760a4a9f7875d843060747cfe9f48795c890a367ca869fc5822a5069" exitCode=0
Oct 31 07:29:17.496142 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:17.496063    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb" event={"ID":"59cf0458-de23-4a5e-884f-4a583e78ff38","Type":"ContainerDied","Data":"bc9f81c3760a4a9f7875d843060747cfe9f48795c890a367ca869fc5822a5069"}
Oct 31 07:29:17.496437 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:17.496417    4913 scope.go:122] "RemoveContainer" containerID="bc9f81c3760a4a9f7875d843060747cfe9f48795c890a367ca869fc5822a5069"
Oct 31 07:29:17.498492 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:17.498458    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svhj9" event={"ID":"b5bca13e-12d2-4673-8223-92f7755eac90","Type":"ContainerStarted","Data":"18ef4ff4e8ba932dac6ca25db1f95b3e692150d6aff1152b94795289f2dee692"}
Oct 31 07:29:17.782944 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:17.782898    4913 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbln7_6121aa17-a37d-47de-af76-08c2aa174568/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:17.964080 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:17.964042    4913 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbln7_6121aa17-a37d-47de-af76-08c2aa174568/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:18.356410 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:18.356368    4913 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbln7_6121aa17-a37d-47de-af76-08c2aa174568/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:18.463102 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:18.463075    4913 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbln7_6121aa17-a37d-47de-af76-08c2aa174568/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:18.504621 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:18.504584    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-qrsxv" event={"ID":"8b210005-d749-4ec8-ad41-bc7afaeb7011","Type":"ContainerStarted","Data":"feeefc4f9f9958212c331503ca081018fff634645d5f05f5e35c8be93e0ab83e"}
Oct 31 07:29:18.506869 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:18.506838    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-ttcjb" event={"ID":"59cf0458-de23-4a5e-884f-4a583e78ff38","Type":"ContainerStarted","Data":"2613074a9013d2974deec53d62806ac4144e5560f8356d693aaac63e4b3a979a"}
Oct 31 07:29:18.508743 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:18.508717    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svhj9" event={"ID":"b5bca13e-12d2-4673-8223-92f7755eac90","Type":"ContainerStarted","Data":"5a100afe42b6fa7695c9742d796e06e7f3ba016a5bea55c0d943ceb6fe8e43e5"}
Oct 31 07:29:18.508834 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:18.508765    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svhj9" event={"ID":"b5bca13e-12d2-4673-8223-92f7755eac90","Type":"ContainerStarted","Data":"aea70735c2c6916c037722774566ed747a54af7ba4ef5e1c6a5957c930a2d1ed"}
Oct 31 07:29:18.568386 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:18.568351    4913 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sbln7_6121aa17-a37d-47de-af76-08c2aa174568/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:20.185892 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.185851    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:20.186435 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.185916    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:20.186435 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.185936    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:20.186435 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.185962    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:20.186435 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.185986    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:20.186435 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.186002    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:20.186435 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:20.186104    4913 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:20.186435 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:20.186177    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle podName:bd4df54c-65e4-4dba-943c-17219493fc36 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:28.186157076 +0000 UTC m=+16.875925356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle") pod "router-default-7f6f75f655-wgvm2" (UID: "bd4df54c-65e4-4dba-943c-17219493fc36") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:20.186435 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:20.186224    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls podName:b0e85bb8-780b-4540-8e9e-19990f22f515 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:28.186214946 +0000 UTC m=+16.875983220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cdwzx" (UID: "b0e85bb8-780b-4540-8e9e-19990f22f515") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:20.188444 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.188417    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"image-registry-7f4d8d5849-jrvx7\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") " pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:20.188578 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.188555    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff383e3f-6228-49ad-bf4e-f43d3eaadf03-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mhwtk\" (UID: \"ff383e3f-6228-49ad-bf4e-f43d3eaadf03\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:20.188814 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.188792    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3adfe0f6-05ab-44f9-8884-06c021c42f5a-metrics-tls\") pod \"dns-default-46cpb\" (UID: \"3adfe0f6-05ab-44f9-8884-06c021c42f5a\") " pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:20.188855 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.188795    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"image-registry-5b69fcb956-87gvh\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") " pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:20.234851 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.234810    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:20.235015 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.234894    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"
Oct 31 07:29:20.250149 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.249940    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:20.251087 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.251041    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:20.522521 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.522455    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk"]
Oct 31 07:29:20.525526 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.525500    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b69fcb956-87gvh"]
Oct 31 07:29:20.529315 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:20.529285    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dd05ff_369c_40dc_9afd_6b54ddbc4f62.slice/crio-02f564905067d6480a4dfae2ace6e589ef9cd3702e7b2e1d3a3f0ae408e24b61 WatchSource:0}: Error finding container 02f564905067d6480a4dfae2ace6e589ef9cd3702e7b2e1d3a3f0ae408e24b61: Status 404 returned error can't find the container with id 02f564905067d6480a4dfae2ace6e589ef9cd3702e7b2e1d3a3f0ae408e24b61
Oct 31 07:29:20.534656 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.534638    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-46cpb"]
Oct 31 07:29:20.539352 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:20.539328    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adfe0f6_05ab_44f9_8884_06c021c42f5a.slice/crio-1d60dec38b81e8987e04761575bc8f8f6e626115fef04a4a2071863c794aa977 WatchSource:0}: Error finding container 1d60dec38b81e8987e04761575bc8f8f6e626115fef04a4a2071863c794aa977: Status 404 returned error can't find the container with id 1d60dec38b81e8987e04761575bc8f8f6e626115fef04a4a2071863c794aa977
Oct 31 07:29:20.549634 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.549613    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4d8d5849-jrvx7"]
Oct 31 07:29:20.551369 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:20.551335    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff9855f_fd75_41e0_82c4_3463f005a54d.slice/crio-871c25bc3de90fddcdf70443c895928e9a29eb9032eb41df4c0cea3bf853c5f2 WatchSource:0}: Error finding container 871c25bc3de90fddcdf70443c895928e9a29eb9032eb41df4c0cea3bf853c5f2: Status 404 returned error can't find the container with id 871c25bc3de90fddcdf70443c895928e9a29eb9032eb41df4c0cea3bf853c5f2
Oct 31 07:29:20.996466 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.996430    4913 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:20.996933 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:20.996908    4913 scope.go:122] "RemoveContainer" containerID="1104fe2e8bf21d08c9e097fd7b74a1b449cdf51197e991d3ec9f97eda867152b"
Oct 31 07:29:20.997152 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:20.997123    4913 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7h5mf_openshift-console-operator(9a1a70f5-55db-4e36-959d-95bae0872fea)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" podUID="9a1a70f5-55db-4e36-959d-95bae0872fea"
Oct 31 07:29:21.521742 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:21.521664    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk" event={"ID":"ff383e3f-6228-49ad-bf4e-f43d3eaadf03","Type":"ContainerStarted","Data":"a17b36bc46d548e64b904c42867d538d440426b64db06200ac05c6024f700acd"}
Oct 31 07:29:21.523717 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:21.523681    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh" event={"ID":"92dd05ff-369c-40dc-9afd-6b54ddbc4f62","Type":"ContainerStarted","Data":"d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f"}
Oct 31 07:29:21.523865 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:21.523726    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh" event={"ID":"92dd05ff-369c-40dc-9afd-6b54ddbc4f62","Type":"ContainerStarted","Data":"02f564905067d6480a4dfae2ace6e589ef9cd3702e7b2e1d3a3f0ae408e24b61"}
Oct 31 07:29:21.526206 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:21.525685    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7" event={"ID":"8ff9855f-fd75-41e0-82c4-3463f005a54d","Type":"ContainerStarted","Data":"5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869"}
Oct 31 07:29:21.526206 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:21.525738    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:21.526206 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:21.525762    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7" event={"ID":"8ff9855f-fd75-41e0-82c4-3463f005a54d","Type":"ContainerStarted","Data":"871c25bc3de90fddcdf70443c895928e9a29eb9032eb41df4c0cea3bf853c5f2"}
Oct 31 07:29:21.527220 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:21.527196    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46cpb" event={"ID":"3adfe0f6-05ab-44f9-8884-06c021c42f5a","Type":"ContainerStarted","Data":"1d60dec38b81e8987e04761575bc8f8f6e626115fef04a4a2071863c794aa977"}
Oct 31 07:29:22.535160 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:22.535071    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk" event={"ID":"ff383e3f-6228-49ad-bf4e-f43d3eaadf03","Type":"ContainerStarted","Data":"1f293ff9bdd7267acd279607c3b6d2550a507efaac2b1917bc3b65473b085696"}
Oct 31 07:29:22.535738 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:22.535176    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mhwtk" event={"ID":"ff383e3f-6228-49ad-bf4e-f43d3eaadf03","Type":"ContainerStarted","Data":"e24c59900164ee162a42885d0da4e8bd2edaa1e3ba6d21a9bbc1939e66aeb288"}
Oct 31 07:29:22.537109 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:22.537083    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46cpb" event={"ID":"3adfe0f6-05ab-44f9-8884-06c021c42f5a","Type":"ContainerStarted","Data":"2ae58fae8910868ae149a910692735e2ed8ada4355ab04914d6cf12934139a89"}
Oct 31 07:29:22.537257 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:22.537240    4913 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:22.537415 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:22.537398    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:22.537462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:22.537449    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46cpb" event={"ID":"3adfe0f6-05ab-44f9-8884-06c021c42f5a","Type":"ContainerStarted","Data":"b64ed393ef1cf88f91ab1db55447a17e329a2363e35cc21f8d3d62f08ba816ec"}
Oct 31 07:29:23.414509 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:23.414472    4913 scope.go:122] "RemoveContainer" containerID="1104fe2e8bf21d08c9e097fd7b74a1b449cdf51197e991d3ec9f97eda867152b"
Oct 31 07:29:23.544230 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:23.544177    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:29:23.544602 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:23.544318    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf" event={"ID":"9a1a70f5-55db-4e36-959d-95bae0872fea","Type":"ContainerStarted","Data":"66a3b285d312bd46c9a8e334d27643a00cdb7884ff968c2135f61d0164622df6"}
Oct 31 07:29:23.544784 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:23.544766    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:23.901670 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:23.901629    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7h5mf"
Oct 31 07:29:27.131111 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:27.131076    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b69fcb956-87gvh"]
Oct 31 07:29:27.131622 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:27.131238    4913 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:28.198773 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.198728    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:28.261518 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.261473    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:28.261518 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.261519    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:28.262116 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.262094    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd4df54c-65e4-4dba-943c-17219493fc36-service-ca-bundle\") pod \"router-default-7f6f75f655-wgvm2\" (UID: \"bd4df54c-65e4-4dba-943c-17219493fc36\") " pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:28.263705 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.263687    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b0e85bb8-780b-4540-8e9e-19990f22f515-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cdwzx\" (UID: \"b0e85bb8-780b-4540-8e9e-19990f22f515\") " pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:28.335297 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.335261    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:28.352150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.352127    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cdwzx"
Oct 31 07:29:28.613235 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.613150    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6f75f655-wgvm2"]
Oct 31 07:29:28.616329 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:28.616294    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4df54c_65e4_4dba_943c_17219493fc36.slice/crio-3a9cd78ea7efa6a07e53418cb932d5f20327fb5648afa45f6d0a142ae6d80418 WatchSource:0}: Error finding container 3a9cd78ea7efa6a07e53418cb932d5f20327fb5648afa45f6d0a142ae6d80418: Status 404 returned error can't find the container with id 3a9cd78ea7efa6a07e53418cb932d5f20327fb5648afa45f6d0a142ae6d80418
Oct 31 07:29:28.634648 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:28.634624    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cdwzx"]
Oct 31 07:29:28.656351 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:28.656319    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e85bb8_780b_4540_8e9e_19990f22f515.slice/crio-26fe880d9ad6df02396eef03af4b3cc9aae88c89a5784f420c243b213c86ec15 WatchSource:0}: Error finding container 26fe880d9ad6df02396eef03af4b3cc9aae88c89a5784f420c243b213c86ec15: Status 404 returned error can't find the container with id 26fe880d9ad6df02396eef03af4b3cc9aae88c89a5784f420c243b213c86ec15
Oct 31 07:29:29.568236 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:29.568132    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6f75f655-wgvm2" event={"ID":"bd4df54c-65e4-4dba-943c-17219493fc36","Type":"ContainerStarted","Data":"5be80cce862f9e53959346ef6b760330e12a4954f63991162ca55c50852ff421"}
Oct 31 07:29:29.568236 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:29.568205    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6f75f655-wgvm2" event={"ID":"bd4df54c-65e4-4dba-943c-17219493fc36","Type":"ContainerStarted","Data":"3a9cd78ea7efa6a07e53418cb932d5f20327fb5648afa45f6d0a142ae6d80418"}
Oct 31 07:29:29.570038 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:29.570016    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cdwzx" event={"ID":"b0e85bb8-780b-4540-8e9e-19990f22f515","Type":"ContainerStarted","Data":"c185e98a5d6c814e0cd0da24c15ea4e3ad177f088fb8a650146802b99d4f6547"}
Oct 31 07:29:29.570154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:29.570059    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cdwzx" event={"ID":"b0e85bb8-780b-4540-8e9e-19990f22f515","Type":"ContainerStarted","Data":"e06cf6473574c20137af3cb89ed01f00fe5e29d8bdf3a57b0697a64c82f35496"}
Oct 31 07:29:29.570154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:29.570077    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cdwzx" event={"ID":"b0e85bb8-780b-4540-8e9e-19990f22f515","Type":"ContainerStarted","Data":"26fe880d9ad6df02396eef03af4b3cc9aae88c89a5784f420c243b213c86ec15"}
Oct 31 07:29:30.336418 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:30.336381    4913 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:30.339317 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:30.339291    4913 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:30.574636 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:30.574593    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:30.575957 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:30.575935    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f6f75f655-wgvm2"
Oct 31 07:29:31.583197 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:31.583150    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cdwzx" event={"ID":"b0e85bb8-780b-4540-8e9e-19990f22f515","Type":"ContainerStarted","Data":"a6fe700679984b546d004927800649de66482f3c246601e049250141305e13e7"}
Oct 31 07:29:33.549914 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:33.549881    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-46cpb"
Oct 31 07:29:37.772362 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:37.772325    4913 patch_prober.go:29] interesting pod/image-registry-7f4d8d5849-jrvx7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 07:29:37.772769 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:37.772386    4913 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7" podUID="8ff9855f-fd75-41e0-82c4-3463f005a54d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:29:38.204522 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:38.204490    4913 patch_prober.go:29] interesting pod/image-registry-5b69fcb956-87gvh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 07:29:38.204673 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:38.204537    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh" podUID="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:29:38.512080 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:38.512011    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sbln7"
Oct 31 07:29:41.360144 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.360106    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9kkbv"]
Oct 31 07:29:41.393995 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.393963    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.421524 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.421466    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 07:29:41.421524 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.421466    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 07:29:41.421708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.421639    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gsfjw\""
Oct 31 07:29:41.459693 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.459636    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 07:29:41.459838 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.459824    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 07:29:41.574459 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574409    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-sys\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.574459 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574443    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91e4782-7ff8-4129-87cc-df5e71fed629-metrics-client-ca\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.574677 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574543    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-wtmp\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.574677 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574585    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-accelerators-collector-config\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.574677 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574650    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-textfile\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.574788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574683    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4t5z\" (UniqueName: \"kubernetes.io/projected/c91e4782-7ff8-4129-87cc-df5e71fed629-kube-api-access-k4t5z\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.574788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574710    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.574788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574778    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-root\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.574890 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.574851    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-tls\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.675920 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.675889    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-root\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676092 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.675939    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-tls\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676092 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.675961    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-sys\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676092 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676001    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91e4782-7ff8-4129-87cc-df5e71fed629-metrics-client-ca\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676092 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676080    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-wtmp\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676328 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676110    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-accelerators-collector-config\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676328 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676142    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-textfile\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676328 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676153    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-root\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676328 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676170    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4t5z\" (UniqueName: \"kubernetes.io/projected/c91e4782-7ff8-4129-87cc-df5e71fed629-kube-api-access-k4t5z\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676328 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676212    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-sys\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676328 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676219    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.676328 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.676293    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-wtmp\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.680255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.677729    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c91e4782-7ff8-4129-87cc-df5e71fed629-metrics-client-ca\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.680255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.679377    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-tls\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.683239 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.680899    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-accelerators-collector-config\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.683474 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.683452    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-textfile\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.683710 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.683688    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c91e4782-7ff8-4129-87cc-df5e71fed629-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.687895 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.687872    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4t5z\" (UniqueName: \"kubernetes.io/projected/c91e4782-7ff8-4129-87cc-df5e71fed629-kube-api-access-k4t5z\") pod \"node-exporter-9kkbv\" (UID: \"c91e4782-7ff8-4129-87cc-df5e71fed629\") " pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.749709 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:41.749675    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9kkbv"
Oct 31 07:29:41.788455 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:29:41.788426    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91e4782_7ff8_4129_87cc_df5e71fed629.slice/crio-44b5fca55ab49bfa87a26b6cc275c3e46d064473a60192bca7cbb3bab5713f9d WatchSource:0}: Error finding container 44b5fca55ab49bfa87a26b6cc275c3e46d064473a60192bca7cbb3bab5713f9d: Status 404 returned error can't find the container with id 44b5fca55ab49bfa87a26b6cc275c3e46d064473a60192bca7cbb3bab5713f9d
Oct 31 07:29:42.541610 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:42.541580    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:29:42.633681 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:42.633613    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9kkbv" event={"ID":"c91e4782-7ff8-4129-87cc-df5e71fed629","Type":"ContainerStarted","Data":"44b5fca55ab49bfa87a26b6cc275c3e46d064473a60192bca7cbb3bab5713f9d"}
Oct 31 07:29:43.638974 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:43.638942    4913 generic.go:356] "Generic (PLEG): container finished" podID="c91e4782-7ff8-4129-87cc-df5e71fed629" containerID="1f38515f63b2fe7069c335935445598d4f289f850f939fdfcf150b14853dfcca" exitCode=0
Oct 31 07:29:43.639399 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:43.639028    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9kkbv" event={"ID":"c91e4782-7ff8-4129-87cc-df5e71fed629","Type":"ContainerDied","Data":"1f38515f63b2fe7069c335935445598d4f289f850f939fdfcf150b14853dfcca"}
Oct 31 07:29:43.974717 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:43.974679    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4d8d5849-jrvx7"]
Oct 31 07:29:44.645098 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:44.645061    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9kkbv" event={"ID":"c91e4782-7ff8-4129-87cc-df5e71fed629","Type":"ContainerStarted","Data":"40e8058f8d8275ee4f90b556419b37d3fd993c579fb7d19a02f90c4ababaa2d3"}
Oct 31 07:29:44.645521 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:44.645113    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9kkbv" event={"ID":"c91e4782-7ff8-4129-87cc-df5e71fed629","Type":"ContainerStarted","Data":"d29a8453ea98f45fb7cfeac112dfd986c49ca473d2ba326e9c746b1ef4058c86"}
Oct 31 07:29:44.666371 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:44.666322    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9kkbv" podStartSLOduration=2.202175321 podStartE2EDuration="3.666307746s" totalImagesPullingTime="1.381121119s" totalInitContainerRuntime="83.011306ms" isStatefulPod=true podCreationTimestamp="2026-10-31 07:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:44.665734728 +0000 UTC m=+33.355503011" watchObservedRunningTime="2026-10-31 07:29:44.666307746 +0000 UTC m=+33.356076034"
Oct 31 07:29:45.346606 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:45.346576    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d5t72"
Oct 31 07:29:48.203032 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:48.203003    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:49.400945 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:49.400907    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:29:52.152062 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.152024    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh" podUID="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" containerName="registry" containerID="cri-o://d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f" gracePeriod=30
Oct 31 07:29:52.407031 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.406977    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:52.464971 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.464941    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") pod \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") "
Oct 31 07:29:52.464971 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.464973    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token\") pod \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") "
Oct 31 07:29:52.465237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.464996    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted\") pod \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") "
Oct 31 07:29:52.465237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.465077    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd\") pod \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") "
Oct 31 07:29:52.465237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.465107    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates\") pod \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") "
Oct 31 07:29:52.465237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.465147    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration\") pod \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") "
Oct 31 07:29:52.465237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.465197    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca\") pod \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") "
Oct 31 07:29:52.465506 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.465248    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets\") pod \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\" (UID: \"92dd05ff-369c-40dc-9afd-6b54ddbc4f62\") "
Oct 31 07:29:52.465735 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.465699    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates" pod "92dd05ff-369c-40dc-9afd-6b54ddbc4f62" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:29:52.465844 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.465809    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca" pod "92dd05ff-369c-40dc-9afd-6b54ddbc4f62" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:29:52.467581 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.467552    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token" pod "92dd05ff-369c-40dc-9afd-6b54ddbc4f62" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:29:52.467695 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.467670    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls" pod "92dd05ff-369c-40dc-9afd-6b54ddbc4f62" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:29:52.467761 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.467709    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd" pod "92dd05ff-369c-40dc-9afd-6b54ddbc4f62" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62"). InnerVolumeSpecName "kube-api-access-ln7bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:29:52.468044 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.468026    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets" pod "92dd05ff-369c-40dc-9afd-6b54ddbc4f62" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:29:52.468157 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.468134    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration" pod "92dd05ff-369c-40dc-9afd-6b54ddbc4f62" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:29:52.472796 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.472773    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted" pod "92dd05ff-369c-40dc-9afd-6b54ddbc4f62" (UID: "92dd05ff-369c-40dc-9afd-6b54ddbc4f62"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:29:52.566374 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.566342    4913 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-installation-pull-secrets\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:29:52.566546 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.566381    4913 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:29:52.566546 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.566396    4913 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-bound-sa-token\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:29:52.566546 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.566408    4913 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-ca-trust-extracted\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:29:52.566546 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.566420    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln7bd\" (UniqueName: \"kubernetes.io/projected/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-kube-api-access-ln7bd\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:29:52.566546 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.566433    4913 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-registry-certificates\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:29:52.566546 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.566462    4913 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-image-registry-private-configuration\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:29:52.566546 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.566474    4913 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92dd05ff-369c-40dc-9afd-6b54ddbc4f62-trusted-ca\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:29:52.676069 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.676032    4913 generic.go:356] "Generic (PLEG): container finished" podID="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" containerID="d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f" exitCode=0
Oct 31 07:29:52.676237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.676083    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh"
Oct 31 07:29:52.676237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.676107    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh" event={"ID":"92dd05ff-369c-40dc-9afd-6b54ddbc4f62","Type":"ContainerDied","Data":"d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f"}
Oct 31 07:29:52.676237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.676156    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b69fcb956-87gvh" event={"ID":"92dd05ff-369c-40dc-9afd-6b54ddbc4f62","Type":"ContainerDied","Data":"02f564905067d6480a4dfae2ace6e589ef9cd3702e7b2e1d3a3f0ae408e24b61"}
Oct 31 07:29:52.676237 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.676198    4913 scope.go:122] "RemoveContainer" containerID="d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f"
Oct 31 07:29:52.684918 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.684896    4913 scope.go:122] "RemoveContainer" containerID="d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f"
Oct 31 07:29:52.685205 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:29:52.685169    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f\": container with ID starting with d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f not found: ID does not exist" containerID="d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f"
Oct 31 07:29:52.685250 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.685216    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f"} err="failed to get container status \"d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f\": rpc error: code = NotFound desc = could not find container \"d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f\": container with ID starting with d50de610a9f24554d476b8eb85b134835d81bffffed864a74823842062a44f1f not found: ID does not exist"
Oct 31 07:29:52.697388 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.697365    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b69fcb956-87gvh"]
Oct 31 07:29:52.703263 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:52.703242    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b69fcb956-87gvh"]
Oct 31 07:29:53.418686 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:29:53.418650    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" path="/var/lib/kubelet/pods/92dd05ff-369c-40dc-9afd-6b54ddbc4f62/volumes"
Oct 31 07:30:08.994250 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:08.994172    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7" podUID="8ff9855f-fd75-41e0-82c4-3463f005a54d" containerName="registry" containerID="cri-o://5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869" gracePeriod=30
Oct 31 07:30:09.236634 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.236607    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:30:09.301639 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.301557    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets\") pod \"8ff9855f-fd75-41e0-82c4-3463f005a54d\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") "
Oct 31 07:30:09.301639 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.301601    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") pod \"8ff9855f-fd75-41e0-82c4-3463f005a54d\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") "
Oct 31 07:30:09.301639 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.301630    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token\") pod \"8ff9855f-fd75-41e0-82c4-3463f005a54d\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") "
Oct 31 07:30:09.301846 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.301802    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca\") pod \"8ff9855f-fd75-41e0-82c4-3463f005a54d\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") "
Oct 31 07:30:09.301894 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.301867    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates\") pod \"8ff9855f-fd75-41e0-82c4-3463f005a54d\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") "
Oct 31 07:30:09.301936 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.301910    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm\") pod \"8ff9855f-fd75-41e0-82c4-3463f005a54d\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") "
Oct 31 07:30:09.301998 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.301945    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted\") pod \"8ff9855f-fd75-41e0-82c4-3463f005a54d\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") "
Oct 31 07:30:09.302050 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.301993    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration\") pod \"8ff9855f-fd75-41e0-82c4-3463f005a54d\" (UID: \"8ff9855f-fd75-41e0-82c4-3463f005a54d\") "
Oct 31 07:30:09.302298 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.302264    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates" pod "8ff9855f-fd75-41e0-82c4-3463f005a54d" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:09.303827 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.302292    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca" pod "8ff9855f-fd75-41e0-82c4-3463f005a54d" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:09.308977 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.304831    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token" pod "8ff9855f-fd75-41e0-82c4-3463f005a54d" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:09.308977 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.304881    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls" pod "8ff9855f-fd75-41e0-82c4-3463f005a54d" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:09.308977 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.305589    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets" pod "8ff9855f-fd75-41e0-82c4-3463f005a54d" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:09.308977 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.305847    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration" pod "8ff9855f-fd75-41e0-82c4-3463f005a54d" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:09.308977 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.306237    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm" pod "8ff9855f-fd75-41e0-82c4-3463f005a54d" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d"). InnerVolumeSpecName "kube-api-access-dw9xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:09.311646 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.311625    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted" pod "8ff9855f-fd75-41e0-82c4-3463f005a54d" (UID: "8ff9855f-fd75-41e0-82c4-3463f005a54d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:30:09.402916 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.402876    4913 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-certificates\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:09.402916 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.402909    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9xm\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-kube-api-access-dw9xm\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:09.402916 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.402919    4913 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ff9855f-fd75-41e0-82c4-3463f005a54d-ca-trust-extracted\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:09.403145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.402931    4913 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-image-registry-private-configuration\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:09.403145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.402941    4913 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ff9855f-fd75-41e0-82c4-3463f005a54d-installation-pull-secrets\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:09.403145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.402952    4913 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-registry-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:09.403145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.402961    4913 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ff9855f-fd75-41e0-82c4-3463f005a54d-bound-sa-token\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:09.403145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.402969    4913 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9855f-fd75-41e0-82c4-3463f005a54d-trusted-ca\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:30:09.747067 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.747031    4913 generic.go:356] "Generic (PLEG): container finished" podID="8ff9855f-fd75-41e0-82c4-3463f005a54d" containerID="5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869" exitCode=0
Oct 31 07:30:09.747269 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.747087    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7"
Oct 31 07:30:09.747269 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.747125    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7" event={"ID":"8ff9855f-fd75-41e0-82c4-3463f005a54d","Type":"ContainerDied","Data":"5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869"}
Oct 31 07:30:09.747269 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.747201    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d8d5849-jrvx7" event={"ID":"8ff9855f-fd75-41e0-82c4-3463f005a54d","Type":"ContainerDied","Data":"871c25bc3de90fddcdf70443c895928e9a29eb9032eb41df4c0cea3bf853c5f2"}
Oct 31 07:30:09.747269 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.747228    4913 scope.go:122] "RemoveContainer" containerID="5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869"
Oct 31 07:30:09.755367 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.755349    4913 scope.go:122] "RemoveContainer" containerID="5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869"
Oct 31 07:30:09.755615 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:30:09.755599    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869\": container with ID starting with 5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869 not found: ID does not exist" containerID="5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869"
Oct 31 07:30:09.755665 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.755623    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869"} err="failed to get container status \"5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869\": rpc error: code = NotFound desc = could not find container \"5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869\": container with ID starting with 5c52442b660151ae6cbdc58d909a825da70c9db2efee4f99913e81a02280b869 not found: ID does not exist"
Oct 31 07:30:09.763345 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.763324    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4d8d5849-jrvx7"]
Oct 31 07:30:09.767046 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:09.767025    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f4d8d5849-jrvx7"]
Oct 31 07:30:11.417865 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:11.417836    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff9855f-fd75-41e0-82c4-3463f005a54d" path="/var/lib/kubelet/pods/8ff9855f-fd75-41e0-82c4-3463f005a54d/volumes"
Oct 31 07:30:19.399886 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:30:19.399844    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:30:26.813057 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:26.812947    4913 generic.go:356] "Generic (PLEG): container finished" podID="87fbf516-95a9-4c2c-a41f-1a053ff98e6b" containerID="ce3afbad417df8792f912ef50b5b4eca9e75ad3a6e45dc06ca490b90541e3b8b" exitCode=0
Oct 31 07:30:26.813057 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:26.813016    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l" event={"ID":"87fbf516-95a9-4c2c-a41f-1a053ff98e6b","Type":"ContainerDied","Data":"ce3afbad417df8792f912ef50b5b4eca9e75ad3a6e45dc06ca490b90541e3b8b"}
Oct 31 07:30:26.813554 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:26.813369    4913 scope.go:122] "RemoveContainer" containerID="ce3afbad417df8792f912ef50b5b4eca9e75ad3a6e45dc06ca490b90541e3b8b"
Oct 31 07:30:27.818262 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:27.818210    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-l546l" event={"ID":"87fbf516-95a9-4c2c-a41f-1a053ff98e6b","Type":"ContainerStarted","Data":"eb32d8cd014bca490dfc51b7effeb9a7d6c66104b4f544ef6c623cf0c8a122fb"}
Oct 31 07:30:36.848361 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:36.848326    4913 generic.go:356] "Generic (PLEG): container finished" podID="e684ad1d-ea28-4d75-aca0-8c12227ffada" containerID="42acaae64f1e406d56d97558dea6441f6cb158c06f6b2bd946f1fb8b0b30c1a2" exitCode=0
Oct 31 07:30:36.848747 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:36.848399    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rjjcf" event={"ID":"e684ad1d-ea28-4d75-aca0-8c12227ffada","Type":"ContainerDied","Data":"42acaae64f1e406d56d97558dea6441f6cb158c06f6b2bd946f1fb8b0b30c1a2"}
Oct 31 07:30:36.848792 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:36.848756    4913 scope.go:122] "RemoveContainer" containerID="42acaae64f1e406d56d97558dea6441f6cb158c06f6b2bd946f1fb8b0b30c1a2"
Oct 31 07:30:37.853523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:30:37.853492    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rjjcf" event={"ID":"e684ad1d-ea28-4d75-aca0-8c12227ffada","Type":"ContainerStarted","Data":"0ac486191409271253e21935e9cb707492b91e617b5e548b3023df0c3d6be19a"}
Oct 31 07:30:49.399786 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:30:49.399744    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:31:19.398654 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:31:19.398613    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:31:49.399902 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:31:49.399857    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:32:19.399931 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:32:19.399891    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:32:49.399483 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:32:49.399444    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:33:09.912247 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.912208    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"]
Oct 31 07:33:09.912788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.912622    4913 cpu_manager.go:414] "containerMap: removing container" podUID="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" containerName="registry"
Oct 31 07:33:09.912788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.912640    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" containerName="registry"
Oct 31 07:33:09.912788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.912654    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8ff9855f-fd75-41e0-82c4-3463f005a54d" containerName="registry"
Oct 31 07:33:09.912788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.912662    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff9855f-fd75-41e0-82c4-3463f005a54d" containerName="registry"
Oct 31 07:33:09.912788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.912758    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="92dd05ff-369c-40dc-9afd-6b54ddbc4f62" containerName="registry"
Oct 31 07:33:09.912788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.912768    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff9855f-fd75-41e0-82c4-3463f005a54d" containerName="registry"
Oct 31 07:33:09.915769 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.915747    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:09.919153 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.919134    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 31 07:33:09.919940 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.919920    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 31 07:33:09.919990 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.919940    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 31 07:33:09.919990 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.919940    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-7rpfl\""
Oct 31 07:33:09.927454 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.927435    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"]
Oct 31 07:33:09.996417 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.996382    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/31bfc015-9db6-4bae-ae33-5396ebed1b7c-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-pkq5q\" (UID: \"31bfc015-9db6-4bae-ae33-5396ebed1b7c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:09.996597 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:09.996472    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97gx9\" (UniqueName: \"kubernetes.io/projected/31bfc015-9db6-4bae-ae33-5396ebed1b7c-kube-api-access-97gx9\") pod \"custom-metrics-autoscaler-operator-c667576c8-pkq5q\" (UID: \"31bfc015-9db6-4bae-ae33-5396ebed1b7c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:10.097268 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:10.097227    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97gx9\" (UniqueName: \"kubernetes.io/projected/31bfc015-9db6-4bae-ae33-5396ebed1b7c-kube-api-access-97gx9\") pod \"custom-metrics-autoscaler-operator-c667576c8-pkq5q\" (UID: \"31bfc015-9db6-4bae-ae33-5396ebed1b7c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:10.097473 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:10.097292    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/31bfc015-9db6-4bae-ae33-5396ebed1b7c-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-pkq5q\" (UID: \"31bfc015-9db6-4bae-ae33-5396ebed1b7c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:10.099696 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:10.099672    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/31bfc015-9db6-4bae-ae33-5396ebed1b7c-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-pkq5q\" (UID: \"31bfc015-9db6-4bae-ae33-5396ebed1b7c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:10.105450 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:10.105424    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97gx9\" (UniqueName: \"kubernetes.io/projected/31bfc015-9db6-4bae-ae33-5396ebed1b7c-kube-api-access-97gx9\") pod \"custom-metrics-autoscaler-operator-c667576c8-pkq5q\" (UID: \"31bfc015-9db6-4bae-ae33-5396ebed1b7c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:10.228584 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:10.228546    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:10.500932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:10.500813    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"]
Oct 31 07:33:10.506338 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:33:10.506286    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bfc015_9db6_4bae_ae33_5396ebed1b7c.slice/crio-3dcaac242977efa6ece2baa62c666c5d07633aaf852af15722015590a777edd6 WatchSource:0}: Error finding container 3dcaac242977efa6ece2baa62c666c5d07633aaf852af15722015590a777edd6: Status 404 returned error can't find the container with id 3dcaac242977efa6ece2baa62c666c5d07633aaf852af15722015590a777edd6
Oct 31 07:33:11.381907 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:11.381875    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q" event={"ID":"31bfc015-9db6-4bae-ae33-5396ebed1b7c","Type":"ContainerStarted","Data":"3dcaac242977efa6ece2baa62c666c5d07633aaf852af15722015590a777edd6"}
Oct 31 07:33:14.396383 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:14.396340    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q" event={"ID":"31bfc015-9db6-4bae-ae33-5396ebed1b7c","Type":"ContainerStarted","Data":"daef6b5477cdf5cfa211cf24ab46ba4cbe2cf63ca68dc1b9f14b7940718fee77"}
Oct 31 07:33:14.396867 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:14.396494    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:14.418140 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:14.417630    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q" podStartSLOduration=1.636726678 podStartE2EDuration="5.417613937s" totalImagesPullingTime="3.780887259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:14.417243293 +0000 UTC m=+243.107011586" watchObservedRunningTime="2026-10-31 07:33:14.417613937 +0000 UTC m=+243.107382229"
Oct 31 07:33:15.070535 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.070504    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-7dc85965f-kkczz"]
Oct 31 07:33:15.073649 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.073632    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.082229 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.082202    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 31 07:33:15.083395 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.083369    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-kkczz"]
Oct 31 07:33:15.108881 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.108862    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Oct 31 07:33:15.109063 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.109044    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-8n9kx\""
Oct 31 07:33:15.131628 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.131605    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.131761 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.131632    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/09fe9dc7-c0c5-4211-bd3b-69871e51c734-cabundle0\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.131761 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.131666    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7kh\" (UniqueName: \"kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-kube-api-access-4d7kh\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.232302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.232259    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7kh\" (UniqueName: \"kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-kube-api-access-4d7kh\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.232488 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.232355    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.232488 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.232380    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/09fe9dc7-c0c5-4211-bd3b-69871e51c734-cabundle0\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.232608 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:15.232505    4913 secret.go:281] references non-existent secret key: ca.crt
Oct 31 07:33:15.232608 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:15.232526    4913 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Oct 31 07:33:15.232608 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:15.232539    4913 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-7dc85965f-kkczz: references non-existent secret key: ca.crt
Oct 31 07:33:15.232608 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:15.232603    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates podName:09fe9dc7-c0c5-4211-bd3b-69871e51c734 nodeName:}" failed. No retries permitted until 2026-10-31 07:33:15.732587533 +0000 UTC m=+244.422355803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates") pod "keda-operator-7dc85965f-kkczz" (UID: "09fe9dc7-c0c5-4211-bd3b-69871e51c734") : references non-existent secret key: ca.crt
Oct 31 07:33:15.233151 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.233126    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/09fe9dc7-c0c5-4211-bd3b-69871e51c734-cabundle0\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.242119 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.242092    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7kh\" (UniqueName: \"kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-kube-api-access-4d7kh\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.736616 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:15.736579    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:15.736991 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:15.736746    4913 secret.go:281] references non-existent secret key: ca.crt
Oct 31 07:33:15.736991 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:15.736775    4913 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Oct 31 07:33:15.736991 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:15.736789    4913 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-7dc85965f-kkczz: references non-existent secret key: ca.crt
Oct 31 07:33:15.736991 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:15.736854    4913 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates podName:09fe9dc7-c0c5-4211-bd3b-69871e51c734 nodeName:}" failed. No retries permitted until 2026-10-31 07:33:16.736835389 +0000 UTC m=+245.426603676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates") pod "keda-operator-7dc85965f-kkczz" (UID: "09fe9dc7-c0c5-4211-bd3b-69871e51c734") : references non-existent secret key: ca.crt
Oct 31 07:33:16.746138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:16.746100    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:16.748736 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:16.748707    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/09fe9dc7-c0c5-4211-bd3b-69871e51c734-certificates\") pod \"keda-operator-7dc85965f-kkczz\" (UID: \"09fe9dc7-c0c5-4211-bd3b-69871e51c734\") " pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:16.886730 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:16.886694    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:17.157035 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:17.156875    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-kkczz"]
Oct 31 07:33:17.159672 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:33:17.159634    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fe9dc7_c0c5_4211_bd3b_69871e51c734.slice/crio-7bf967cda23fe61decba19138cb1ff2f6b8894bd00f33de922e021dcfd224489 WatchSource:0}: Error finding container 7bf967cda23fe61decba19138cb1ff2f6b8894bd00f33de922e021dcfd224489: Status 404 returned error can't find the container with id 7bf967cda23fe61decba19138cb1ff2f6b8894bd00f33de922e021dcfd224489
Oct 31 07:33:17.409119 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:17.409041    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-kkczz" event={"ID":"09fe9dc7-c0c5-4211-bd3b-69871e51c734","Type":"ContainerStarted","Data":"7bf967cda23fe61decba19138cb1ff2f6b8894bd00f33de922e021dcfd224489"}
Oct 31 07:33:19.403402 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:19.403354    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:33:21.426647 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:21.426610    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-kkczz" event={"ID":"09fe9dc7-c0c5-4211-bd3b-69871e51c734","Type":"ContainerStarted","Data":"9d34f5702c6b37d14be80f715200ff55cb2cf27fccd459bd8b1d77796ce90f0e"}
Oct 31 07:33:21.427005 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:21.426682    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:21.442272 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:21.442218    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-operator-7dc85965f-kkczz" podStartSLOduration=2.64875958 podStartE2EDuration="6.442204158s" totalImagesPullingTime="3.793444578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:21.441628725 +0000 UTC m=+250.131397033" watchObservedRunningTime="2026-10-31 07:33:21.442204158 +0000 UTC m=+250.131972444"
Oct 31 07:33:35.403401 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:35.403366    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-pkq5q"
Oct 31 07:33:42.433279 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:33:42.433247    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-7dc85965f-kkczz"
Oct 31 07:33:49.399161 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:33:49.399124    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:34:11.434254 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:11.434224    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:34:11.434712 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:11.434524    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:34:19.399857 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:34:19.399815    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:34:37.384495 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.384456    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-6vdbd"]
Oct 31 07:34:37.387848 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.387826    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:37.397423 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.397380    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qkxtn\""
Oct 31 07:34:37.397955 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.397916    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Oct 31 07:34:37.398219 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.398163    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 07:34:37.398319 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.398267    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 07:34:37.399274 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.399250    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-6vdbd"]
Oct 31 07:34:37.466128 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.466098    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v7pq\" (UniqueName: \"kubernetes.io/projected/d21b6d95-3c3a-4631-ad46-caba5da2cf35-kube-api-access-7v7pq\") pod \"kserve-controller-manager-76d8b77fbb-6vdbd\" (UID: \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\") " pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:37.466294 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.466150    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21b6d95-3c3a-4631-ad46-caba5da2cf35-cert\") pod \"kserve-controller-manager-76d8b77fbb-6vdbd\" (UID: \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\") " pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:37.567114 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.567078    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21b6d95-3c3a-4631-ad46-caba5da2cf35-cert\") pod \"kserve-controller-manager-76d8b77fbb-6vdbd\" (UID: \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\") " pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:37.567288 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.567247    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v7pq\" (UniqueName: \"kubernetes.io/projected/d21b6d95-3c3a-4631-ad46-caba5da2cf35-kube-api-access-7v7pq\") pod \"kserve-controller-manager-76d8b77fbb-6vdbd\" (UID: \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\") " pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:37.569531 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.569497    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21b6d95-3c3a-4631-ad46-caba5da2cf35-cert\") pod \"kserve-controller-manager-76d8b77fbb-6vdbd\" (UID: \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\") " pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:37.574986 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.574957    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v7pq\" (UniqueName: \"kubernetes.io/projected/d21b6d95-3c3a-4631-ad46-caba5da2cf35-kube-api-access-7v7pq\") pod \"kserve-controller-manager-76d8b77fbb-6vdbd\" (UID: \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\") " pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:37.700839 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.700801    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:37.970348 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.970282    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-6vdbd"]
Oct 31 07:34:37.972952 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:34:37.972913    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21b6d95_3c3a_4631_ad46_caba5da2cf35.slice/crio-73eed1e379aa8c546b65bcf66998064d1abebd88ce9e450546de90f8bceaf47c WatchSource:0}: Error finding container 73eed1e379aa8c546b65bcf66998064d1abebd88ce9e450546de90f8bceaf47c: Status 404 returned error can't find the container with id 73eed1e379aa8c546b65bcf66998064d1abebd88ce9e450546de90f8bceaf47c
Oct 31 07:34:37.974284 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:37.974264    4913 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:34:38.722980 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:38.722946    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd" event={"ID":"d21b6d95-3c3a-4631-ad46-caba5da2cf35","Type":"ContainerStarted","Data":"73eed1e379aa8c546b65bcf66998064d1abebd88ce9e450546de90f8bceaf47c"}
Oct 31 07:34:41.737011 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:41.736976    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd" event={"ID":"d21b6d95-3c3a-4631-ad46-caba5da2cf35","Type":"ContainerStarted","Data":"987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7"}
Oct 31 07:34:41.737450 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:41.737132    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:34:41.753198 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:34:41.753141    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd" podStartSLOduration=2.010274299 podStartE2EDuration="4.753127868s" totalImagesPullingTime="2.742853569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:34:41.751815008 +0000 UTC m=+330.441583300" watchObservedRunningTime="2026-10-31 07:34:41.753127868 +0000 UTC m=+330.442896159"
Oct 31 07:34:49.399873 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:34:49.399837    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:35:12.746480 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:12.746447    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:35:19.399729 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:35:19.399687    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:35:21.778097 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:21.778058    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-6vdbd"]
Oct 31 07:35:21.778557 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:21.778301    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd" podUID="d21b6d95-3c3a-4631-ad46-caba5da2cf35" containerName="manager" containerID="cri-o://987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7" gracePeriod=10
Oct 31 07:35:21.799570 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:21.799543    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-mq67h"]
Oct 31 07:35:21.802745 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:21.802728    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:21.812221 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:21.812177    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-mq67h"]
Oct 31 07:35:21.920305 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:21.920256    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c5269f9-4eef-43d7-a2a9-7d134c7e9bae-cert\") pod \"kserve-controller-manager-76d8b77fbb-mq67h\" (UID: \"7c5269f9-4eef-43d7-a2a9-7d134c7e9bae\") " pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:21.920483 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:21.920381    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8f2q\" (UniqueName: \"kubernetes.io/projected/7c5269f9-4eef-43d7-a2a9-7d134c7e9bae-kube-api-access-g8f2q\") pod \"kserve-controller-manager-76d8b77fbb-mq67h\" (UID: \"7c5269f9-4eef-43d7-a2a9-7d134c7e9bae\") " pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:22.015994 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.015967    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:35:22.021551 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.021526    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8f2q\" (UniqueName: \"kubernetes.io/projected/7c5269f9-4eef-43d7-a2a9-7d134c7e9bae-kube-api-access-g8f2q\") pod \"kserve-controller-manager-76d8b77fbb-mq67h\" (UID: \"7c5269f9-4eef-43d7-a2a9-7d134c7e9bae\") " pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:22.021680 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.021620    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c5269f9-4eef-43d7-a2a9-7d134c7e9bae-cert\") pod \"kserve-controller-manager-76d8b77fbb-mq67h\" (UID: \"7c5269f9-4eef-43d7-a2a9-7d134c7e9bae\") " pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:22.023899 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.023876    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c5269f9-4eef-43d7-a2a9-7d134c7e9bae-cert\") pod \"kserve-controller-manager-76d8b77fbb-mq67h\" (UID: \"7c5269f9-4eef-43d7-a2a9-7d134c7e9bae\") " pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:22.029626 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.029568    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8f2q\" (UniqueName: \"kubernetes.io/projected/7c5269f9-4eef-43d7-a2a9-7d134c7e9bae-kube-api-access-g8f2q\") pod \"kserve-controller-manager-76d8b77fbb-mq67h\" (UID: \"7c5269f9-4eef-43d7-a2a9-7d134c7e9bae\") " pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:22.122326 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.122292    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21b6d95-3c3a-4631-ad46-caba5da2cf35-kube-api-access-7v7pq\" (UniqueName: \"kubernetes.io/projected/d21b6d95-3c3a-4631-ad46-caba5da2cf35-kube-api-access-7v7pq\") pod \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\" (UID: \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\") "
Oct 31 07:35:22.122489 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.122331    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21b6d95-3c3a-4631-ad46-caba5da2cf35-cert\" (UniqueName: \"kubernetes.io/secret/d21b6d95-3c3a-4631-ad46-caba5da2cf35-cert\") pod \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\" (UID: \"d21b6d95-3c3a-4631-ad46-caba5da2cf35\") "
Oct 31 07:35:22.124513 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.124478    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21b6d95-3c3a-4631-ad46-caba5da2cf35-kube-api-access-7v7pq" pod "d21b6d95-3c3a-4631-ad46-caba5da2cf35" (UID: "d21b6d95-3c3a-4631-ad46-caba5da2cf35"). InnerVolumeSpecName "kube-api-access-7v7pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:35:22.124620 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.124546    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21b6d95-3c3a-4631-ad46-caba5da2cf35-cert" pod "d21b6d95-3c3a-4631-ad46-caba5da2cf35" (UID: "d21b6d95-3c3a-4631-ad46-caba5da2cf35"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:35:22.157842 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.157817    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:22.223749 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.223715    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v7pq\" (UniqueName: \"kubernetes.io/projected/d21b6d95-3c3a-4631-ad46-caba5da2cf35-kube-api-access-7v7pq\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:35:22.223749 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.223746    4913 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d21b6d95-3c3a-4631-ad46-caba5da2cf35-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:35:22.425392 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.425364    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-mq67h"]
Oct 31 07:35:22.427779 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:35:22.427752    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5269f9_4eef_43d7_a2a9_7d134c7e9bae.slice/crio-c4abaddd985d738801400bf82d5ab749455998fe6a73bcd5af4b47f20546f6d4 WatchSource:0}: Error finding container c4abaddd985d738801400bf82d5ab749455998fe6a73bcd5af4b47f20546f6d4: Status 404 returned error can't find the container with id c4abaddd985d738801400bf82d5ab749455998fe6a73bcd5af4b47f20546f6d4
Oct 31 07:35:22.896397 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.896357    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h" event={"ID":"7c5269f9-4eef-43d7-a2a9-7d134c7e9bae","Type":"ContainerStarted","Data":"c4abaddd985d738801400bf82d5ab749455998fe6a73bcd5af4b47f20546f6d4"}
Oct 31 07:35:22.897610 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.897585    4913 generic.go:356] "Generic (PLEG): container finished" podID="d21b6d95-3c3a-4631-ad46-caba5da2cf35" containerID="987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7" exitCode=0
Oct 31 07:35:22.897669 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.897638    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd"
Oct 31 07:35:22.897669 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.897656    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd" event={"ID":"d21b6d95-3c3a-4631-ad46-caba5da2cf35","Type":"ContainerDied","Data":"987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7"}
Oct 31 07:35:22.897737 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.897691    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-6vdbd" event={"ID":"d21b6d95-3c3a-4631-ad46-caba5da2cf35","Type":"ContainerDied","Data":"73eed1e379aa8c546b65bcf66998064d1abebd88ce9e450546de90f8bceaf47c"}
Oct 31 07:35:22.897737 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.897713    4913 scope.go:122] "RemoveContainer" containerID="987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7"
Oct 31 07:35:22.907293 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.907269    4913 scope.go:122] "RemoveContainer" containerID="987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7"
Oct 31 07:35:22.908052 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:35:22.907572    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7\": container with ID starting with 987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7 not found: ID does not exist" containerID="987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7"
Oct 31 07:35:22.908052 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.907609    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7"} err="failed to get container status \"987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7\": rpc error: code = NotFound desc = could not find container \"987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7\": container with ID starting with 987d87a47307eebd1beac0952c33612924e5620ca58241e0704001dfb3c8b0f7 not found: ID does not exist"
Oct 31 07:35:22.924663 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.924637    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-6vdbd"]
Oct 31 07:35:22.926510 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:22.926490    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-6vdbd"]
Oct 31 07:35:23.419637 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:23.419603    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21b6d95-3c3a-4631-ad46-caba5da2cf35" path="/var/lib/kubelet/pods/d21b6d95-3c3a-4631-ad46-caba5da2cf35/volumes"
Oct 31 07:35:23.903587 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:23.903546    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h" event={"ID":"7c5269f9-4eef-43d7-a2a9-7d134c7e9bae","Type":"ContainerStarted","Data":"4f2a5bab5d8a182f627f1658c8a3626572d333af92cdd619094fec8f359892af"}
Oct 31 07:35:23.904071 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:23.903657    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:35:23.918426 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:23.918374    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h" podStartSLOduration=2.450159837 podStartE2EDuration="2.918355711s" totalImagesPullingTime="468.195874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:35:23.918337668 +0000 UTC m=+372.608105983" watchObservedRunningTime="2026-10-31 07:35:23.918355711 +0000 UTC m=+372.608124003"
Oct 31 07:35:49.398620 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:35:49.398584    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:35:54.912093 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:35:54.912017    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-76d8b77fbb-mq67h"
Oct 31 07:36:19.399788 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:36:19.399746    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:36:49.399003 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:36:49.398963    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:37:19.399195 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:37:19.399141    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:37:32.661870 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.661833    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"]
Oct 31 07:37:32.662365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.662324    4913 cpu_manager.go:414] "containerMap: removing container" podUID="d21b6d95-3c3a-4631-ad46-caba5da2cf35" containerName="manager"
Oct 31 07:37:32.662365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.662342    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21b6d95-3c3a-4631-ad46-caba5da2cf35" containerName="manager"
Oct 31 07:37:32.662478 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.662432    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21b6d95-3c3a-4631-ad46-caba5da2cf35" containerName="manager"
Oct 31 07:37:32.664906 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.664885    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:32.667946 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.667924    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Oct 31 07:37:32.668742 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.668727    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c4r7k\""
Oct 31 07:37:32.672103 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.672080    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"]
Oct 31 07:37:32.696493 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.696464    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjxj\" (UniqueName: \"kubernetes.io/projected/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-kube-api-access-qjjxj\") pod \"seaweedfs-tls-custom-64cc6669cb-dp9qm\" (UID: \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:32.696612 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.696519    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-data\") pod \"seaweedfs-tls-custom-64cc6669cb-dp9qm\" (UID: \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:32.797812 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.797774    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjxj\" (UniqueName: \"kubernetes.io/projected/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-kube-api-access-qjjxj\") pod \"seaweedfs-tls-custom-64cc6669cb-dp9qm\" (UID: \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:32.798005 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.797838    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-data\") pod \"seaweedfs-tls-custom-64cc6669cb-dp9qm\" (UID: \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:32.798229 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.798212    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-data\") pod \"seaweedfs-tls-custom-64cc6669cb-dp9qm\" (UID: \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:32.806644 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.806626    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjxj\" (UniqueName: \"kubernetes.io/projected/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-kube-api-access-qjjxj\") pod \"seaweedfs-tls-custom-64cc6669cb-dp9qm\" (UID: \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\") " pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:32.976412 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:32.976376    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:33.243560 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:33.243490    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"]
Oct 31 07:37:33.246788 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:37:33.246755    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f04550_388e_4e5a_9fa5_95ebd3fa90ec.slice/crio-f26c846b20424475b492c4cab321981dee938c9ab51f9c0a02b58869cad81c8c WatchSource:0}: Error finding container f26c846b20424475b492c4cab321981dee938c9ab51f9c0a02b58869cad81c8c: Status 404 returned error can't find the container with id f26c846b20424475b492c4cab321981dee938c9ab51f9c0a02b58869cad81c8c
Oct 31 07:37:33.395020 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:33.394984    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm" event={"ID":"08f04550-388e-4e5a-9fa5-95ebd3fa90ec","Type":"ContainerStarted","Data":"f26c846b20424475b492c4cab321981dee938c9ab51f9c0a02b58869cad81c8c"}
Oct 31 07:37:36.414418 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:36.414383    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm" event={"ID":"08f04550-388e-4e5a-9fa5-95ebd3fa90ec","Type":"ContainerStarted","Data":"92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a"}
Oct 31 07:37:36.430585 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:36.430531    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm" podStartSLOduration=2.081185073 podStartE2EDuration="4.430517663s" totalImagesPullingTime="2.34933259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:36.42968032 +0000 UTC m=+505.119448612" watchObservedRunningTime="2026-10-31 07:37:36.430517663 +0000 UTC m=+505.120285955"
Oct 31 07:37:37.577126 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:37.577090    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"]
Oct 31 07:37:38.421993 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:38.421953    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm" podUID="08f04550-388e-4e5a-9fa5-95ebd3fa90ec" containerName="seaweedfs-tls-custom" containerID="cri-o://92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a" gracePeriod=30
Oct 31 07:37:39.661433 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.661404    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:39.704208 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.704112    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"]
Oct 31 07:37:39.704466 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.704451    4913 cpu_manager.go:414] "containerMap: removing container" podUID="08f04550-388e-4e5a-9fa5-95ebd3fa90ec" containerName="seaweedfs-tls-custom"
Oct 31 07:37:39.704509 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.704468    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f04550-388e-4e5a-9fa5-95ebd3fa90ec" containerName="seaweedfs-tls-custom"
Oct 31 07:37:39.704542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.704520    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f04550-388e-4e5a-9fa5-95ebd3fa90ec" containerName="seaweedfs-tls-custom"
Oct 31 07:37:39.707470 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.707441    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.731848 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.731826    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Oct 31 07:37:39.737338 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.737311    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"]
Oct 31 07:37:39.755505 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.755472    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-data\" (UniqueName: \"kubernetes.io/empty-dir/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-data\") pod \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\" (UID: \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\") "
Oct 31 07:37:39.755661 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.755526    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-kube-api-access-qjjxj\" (UniqueName: \"kubernetes.io/projected/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-kube-api-access-qjjxj\") pod \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\" (UID: \"08f04550-388e-4e5a-9fa5-95ebd3fa90ec\") "
Oct 31 07:37:39.755661 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.755612    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e407da07-e3c6-44f1-979a-fb525bf8c30b-data\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.755747 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.755687    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e407da07-e3c6-44f1-979a-fb525bf8c30b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.755747 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.755715    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmd7\" (UniqueName: \"kubernetes.io/projected/e407da07-e3c6-44f1-979a-fb525bf8c30b-kube-api-access-wmmd7\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.756753 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.756728    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-data" pod "08f04550-388e-4e5a-9fa5-95ebd3fa90ec" (UID: "08f04550-388e-4e5a-9fa5-95ebd3fa90ec"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:37:39.757589 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.757564    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-kube-api-access-qjjxj" pod "08f04550-388e-4e5a-9fa5-95ebd3fa90ec" (UID: "08f04550-388e-4e5a-9fa5-95ebd3fa90ec"). InnerVolumeSpecName "kube-api-access-qjjxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:37:39.856655 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.856605    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e407da07-e3c6-44f1-979a-fb525bf8c30b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.856655 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.856655    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmd7\" (UniqueName: \"kubernetes.io/projected/e407da07-e3c6-44f1-979a-fb525bf8c30b-kube-api-access-wmmd7\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.856866 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.856711    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e407da07-e3c6-44f1-979a-fb525bf8c30b-data\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.856866 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.856739    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjxj\" (UniqueName: \"kubernetes.io/projected/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-kube-api-access-qjjxj\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:37:39.856866 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.856752    4913 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/08f04550-388e-4e5a-9fa5-95ebd3fa90ec-data\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:37:39.857129 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.857113    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e407da07-e3c6-44f1-979a-fb525bf8c30b-data\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.859101 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.859081    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e407da07-e3c6-44f1-979a-fb525bf8c30b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:39.865199 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:39.865159    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmd7\" (UniqueName: \"kubernetes.io/projected/e407da07-e3c6-44f1-979a-fb525bf8c30b-kube-api-access-wmmd7\") pod \"seaweedfs-tls-custom-685c9f986c-6qbvb\" (UID: \"e407da07-e3c6-44f1-979a-fb525bf8c30b\") " pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:40.020447 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.020364    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"
Oct 31 07:37:40.289693 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.289668    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-685c9f986c-6qbvb"]
Oct 31 07:37:40.291977 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:37:40.291946    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode407da07_e3c6_44f1_979a_fb525bf8c30b.slice/crio-f109a5f775ef2a3eba0eaf8378bce537c94d5619d94a4f06875b284a443eb6c5 WatchSource:0}: Error finding container f109a5f775ef2a3eba0eaf8378bce537c94d5619d94a4f06875b284a443eb6c5: Status 404 returned error can't find the container with id f109a5f775ef2a3eba0eaf8378bce537c94d5619d94a4f06875b284a443eb6c5
Oct 31 07:37:40.430896 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.430853    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb" event={"ID":"e407da07-e3c6-44f1-979a-fb525bf8c30b","Type":"ContainerStarted","Data":"f109a5f775ef2a3eba0eaf8378bce537c94d5619d94a4f06875b284a443eb6c5"}
Oct 31 07:37:40.432083 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.432053    4913 generic.go:356] "Generic (PLEG): container finished" podID="08f04550-388e-4e5a-9fa5-95ebd3fa90ec" containerID="92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a" exitCode=0
Oct 31 07:37:40.432214 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.432090    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm" event={"ID":"08f04550-388e-4e5a-9fa5-95ebd3fa90ec","Type":"ContainerDied","Data":"92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a"}
Oct 31 07:37:40.432214 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.432103    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"
Oct 31 07:37:40.432214 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.432123    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm" event={"ID":"08f04550-388e-4e5a-9fa5-95ebd3fa90ec","Type":"ContainerDied","Data":"f26c846b20424475b492c4cab321981dee938c9ab51f9c0a02b58869cad81c8c"}
Oct 31 07:37:40.432214 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.432149    4913 scope.go:122] "RemoveContainer" containerID="92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a"
Oct 31 07:37:40.449281 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.449262    4913 scope.go:122] "RemoveContainer" containerID="92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a"
Oct 31 07:37:40.449814 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:37:40.449792    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a\": container with ID starting with 92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a not found: ID does not exist" containerID="92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a"
Oct 31 07:37:40.449908 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.449820    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a"} err="failed to get container status \"92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a\": rpc error: code = NotFound desc = could not find container \"92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a\": container with ID starting with 92e185cd893e2bf08a1dbc4e3e112b25fb9b86ce3b53e3082fee1bb074b2c37a not found: ID does not exist"
Oct 31 07:37:40.456722 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.456701    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"]
Oct 31 07:37:40.459827 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:40.459805    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-64cc6669cb-dp9qm"]
Oct 31 07:37:41.424520 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:41.424485    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f04550-388e-4e5a-9fa5-95ebd3fa90ec" path="/var/lib/kubelet/pods/08f04550-388e-4e5a-9fa5-95ebd3fa90ec/volumes"
Oct 31 07:37:41.438574 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:41.438537    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb" event={"ID":"e407da07-e3c6-44f1-979a-fb525bf8c30b","Type":"ContainerStarted","Data":"bce9d933131fce7a4d7c898df5ee673b11858138faf4c53a1f8d267b34fd26cd"}
Oct 31 07:37:41.454571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:41.454514    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-685c9f986c-6qbvb" podStartSLOduration=2.139175289 podStartE2EDuration="2.454496725s" totalImagesPullingTime="315.321436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:41.453853949 +0000 UTC m=+510.143622242" watchObservedRunningTime="2026-10-31 07:37:41.454496725 +0000 UTC m=+510.144265020"
Oct 31 07:37:49.435009 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:37:49.434943    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:37:52.541800 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.541762    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-nq7pq"]
Oct 31 07:37:52.545297 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.545274    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.553407 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.553381    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Oct 31 07:37:52.553716 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.553694    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Oct 31 07:37:52.554784 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.554762    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-nq7pq"]
Oct 31 07:37:52.665204 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.665146    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a012adb2-0a1b-47d4-9eb7-115261749051-data\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.665368 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.665283    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a012adb2-0a1b-47d4-9eb7-115261749051-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.665406 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.665371    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wzg6\" (UniqueName: \"kubernetes.io/projected/a012adb2-0a1b-47d4-9eb7-115261749051-kube-api-access-6wzg6\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.766331 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.766293    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wzg6\" (UniqueName: \"kubernetes.io/projected/a012adb2-0a1b-47d4-9eb7-115261749051-kube-api-access-6wzg6\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.766446 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.766349    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a012adb2-0a1b-47d4-9eb7-115261749051-data\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.766446 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.766388    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a012adb2-0a1b-47d4-9eb7-115261749051-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.766757 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.766734    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a012adb2-0a1b-47d4-9eb7-115261749051-data\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.768746 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.768728    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a012adb2-0a1b-47d4-9eb7-115261749051-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.774986 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.774960    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wzg6\" (UniqueName: \"kubernetes.io/projected/a012adb2-0a1b-47d4-9eb7-115261749051-kube-api-access-6wzg6\") pod \"seaweedfs-tls-serving-5c74974785-nq7pq\" (UID: \"a012adb2-0a1b-47d4-9eb7-115261749051\") " pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:52.857144 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:52.857064    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq"
Oct 31 07:37:53.124938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:53.124857    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-nq7pq"]
Oct 31 07:37:53.127880 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:37:53.127853    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda012adb2_0a1b_47d4_9eb7_115261749051.slice/crio-1ddf7768f745a364e3025365ec62f65a476b4c52c277b4f9ea9716f78685fad8 WatchSource:0}: Error finding container 1ddf7768f745a364e3025365ec62f65a476b4c52c277b4f9ea9716f78685fad8: Status 404 returned error can't find the container with id 1ddf7768f745a364e3025365ec62f65a476b4c52c277b4f9ea9716f78685fad8
Oct 31 07:37:53.485214 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:53.485171    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq" event={"ID":"a012adb2-0a1b-47d4-9eb7-115261749051","Type":"ContainerStarted","Data":"1ddf7768f745a364e3025365ec62f65a476b4c52c277b4f9ea9716f78685fad8"}
Oct 31 07:37:54.490108 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:54.490073    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq" event={"ID":"a012adb2-0a1b-47d4-9eb7-115261749051","Type":"ContainerStarted","Data":"11e37a5926e8d07350c1836fee3c6882d54cdba51706dc5f4a2cb6189db030f5"}
Oct 31 07:37:54.506017 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:37:54.505973    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-5c74974785-nq7pq" podStartSLOduration=2.185184474 podStartE2EDuration="2.505960795s" totalImagesPullingTime="320.776321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:54.505576448 +0000 UTC m=+523.195344740" watchObservedRunningTime="2026-10-31 07:37:54.505960795 +0000 UTC m=+523.195729087"
Oct 31 07:38:19.400243 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:38:19.400209    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:38:23.146273 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.146240    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"]
Oct 31 07:38:23.151588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.151546    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.155017 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.154993    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Oct 31 07:38:23.155134 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.155006    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 07:38:23.155134 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.155076    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:38:23.155973 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.155958    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:38:23.156028 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.155976    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:38:23.156028 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.155991    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:38:23.156028 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.155994    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Oct 31 07:38:23.159875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.159845    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"]
Oct 31 07:38:23.220481 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.220447    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.220652 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.220498    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.220652 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.220588    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.220652 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.220629    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88z2g\" (UniqueName: \"kubernetes.io/projected/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-api-access-88z2g\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.220782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.220655    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d963654-fdde-4615-98f1-a8fdac7dfc9e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.320983 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.320944    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.320983 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.320986    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.321265 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.321007    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88z2g\" (UniqueName: \"kubernetes.io/projected/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-api-access-88z2g\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.321265 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.321044    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d963654-fdde-4615-98f1-a8fdac7dfc9e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.321265 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.321075    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.321452 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.321429    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.321795 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.321776    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.321837 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.321776    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.323487 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.323466    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d963654-fdde-4615-98f1-a8fdac7dfc9e-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.329019 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.328994    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88z2g\" (UniqueName: \"kubernetes.io/projected/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-api-access-88z2g\") pod \"isvc-sklearn-batcher-predictor-7468d78d5-bvkcz\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.473636 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.473599    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:23.750054 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:23.750024    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"]
Oct 31 07:38:23.752525 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:38:23.752495    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d963654_fdde_4615_98f1_a8fdac7dfc9e.slice/crio-6c33ccc16e16d2ff7927cb1256bcaed25b64993af4451443510cc8588727d790 WatchSource:0}: Error finding container 6c33ccc16e16d2ff7927cb1256bcaed25b64993af4451443510cc8588727d790: Status 404 returned error can't find the container with id 6c33ccc16e16d2ff7927cb1256bcaed25b64993af4451443510cc8588727d790
Oct 31 07:38:24.612522 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:24.612482    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerStarted","Data":"6c33ccc16e16d2ff7927cb1256bcaed25b64993af4451443510cc8588727d790"}
Oct 31 07:38:27.627759 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:27.627721    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerStarted","Data":"ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835"}
Oct 31 07:38:31.643588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:31.643547    4913 generic.go:356] "Generic (PLEG): container finished" podID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerID="ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835" exitCode=0
Oct 31 07:38:31.644086 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:31.643586    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerDied","Data":"ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835"}
Oct 31 07:38:45.720745 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:45.720651    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerStarted","Data":"e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506"}
Oct 31 07:38:47.733216 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:47.733169    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerStarted","Data":"030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7"}
Oct 31 07:38:49.401881 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:38:49.401844    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:38:51.752204 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:51.752152    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerStarted","Data":"bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946"}
Oct 31 07:38:51.752692 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:51.752379    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:51.752692 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:51.752413    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:51.753914 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:51.753886    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:38:51.772108 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:51.772059    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podStartSLOduration=2.393065466 podStartE2EDuration="28.772044136s" totalImagesPullingTime="22.341438584s" totalInitContainerRuntime="4.037540086s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:23 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:51.770498311 +0000 UTC m=+580.460266603" watchObservedRunningTime="2026-10-31 07:38:51.772044136 +0000 UTC m=+580.461812491"
Oct 31 07:38:52.757051 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:52.757019    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:52.757630 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:52.757029    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:38:52.758321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:52.758297    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:38:52.761097 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:52.761079    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:38:53.761868 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:53.761828    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:38:53.762329 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:53.762162    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:38:54.765741 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:54.765703    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:38:54.766159 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:38:54.765949    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:39:04.766526 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:04.766471    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:39:04.767028 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:04.766873    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:39:11.463951 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:11.463923    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:39:11.464487 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:11.464268    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:39:14.766095 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:14.766042    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:39:14.766619 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:14.766507    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:39:19.400255 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:39:19.400218    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:39:24.766506 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:24.766455    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:39:24.767007 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:24.766978    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:39:34.765954 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:34.765899    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:39:34.766397 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:34.766349    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:39:44.765925 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:44.765875    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:39:44.766441 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:44.766274    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:39:49.399926 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:39:49.399887    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:39:54.766514 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:54.766471    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:39:54.767034 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:39:54.766938    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:40:04.767109 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:04.767075    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:40:04.767528 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:04.767410    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:40:18.654784 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:18.654750    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"]
Oct 31 07:40:18.655279 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:18.655009    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" containerID="cri-o://e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506" gracePeriod=30
Oct 31 07:40:18.655279 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:18.655043    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" containerID="cri-o://bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946" gracePeriod=30
Oct 31 07:40:18.655279 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:18.655075    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy" containerID="cri-o://030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7" gracePeriod=30
Oct 31 07:40:19.118269 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:19.118237    4913 generic.go:356] "Generic (PLEG): container finished" podID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerID="030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7" exitCode=0
Oct 31 07:40:19.118436 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:19.118299    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerDied","Data":"030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7"}
Oct 31 07:40:19.400007 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:40:19.399926    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:40:20.190261 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.190227    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"]
Oct 31 07:40:20.193788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.193771    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.198638 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.198612    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:40:20.198770 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.198655    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-primary-fa781c-kube-rbac-proxy-sar-config\""
Oct 31 07:40:20.198770 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.198679    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:40:20.198770 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.198691    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-primary-fa781c-predictor-serving-cert\""
Oct 31 07:40:20.198770 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.198700    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:40:20.198770 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.198679    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:40:20.198770 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.198679    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:40:20.203234 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.203212    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"]
Oct 31 07:40:20.283733 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.283700    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnmr\" (UniqueName: \"kubernetes.io/projected/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-api-access-ftnmr\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.283733 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.283736    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.283938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.283760    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/032d76f7-5c87-4f22-b127-76d386e60b2c-proxy-tls\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.283938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.283834    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-cabundle-cert\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.283938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.283908    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032d76f7-5c87-4f22-b127-76d386e60b2c-kserve-provision-location\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.384681 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.384647    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/032d76f7-5c87-4f22-b127-76d386e60b2c-proxy-tls\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.384681 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.384680    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-cabundle-cert\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.384911 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.384720    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032d76f7-5c87-4f22-b127-76d386e60b2c-kserve-provision-location\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.384911 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.384789    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnmr\" (UniqueName: \"kubernetes.io/projected/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-api-access-ftnmr\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.384911 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.384816    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.385250 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.385228    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032d76f7-5c87-4f22-b127-76d386e60b2c-kserve-provision-location\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.385564 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.385543    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-cabundle-cert\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.385612 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.385543    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.387097 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.387079    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/032d76f7-5c87-4f22-b127-76d386e60b2c-proxy-tls\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.391916 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.391895    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnmr\" (UniqueName: \"kubernetes.io/projected/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-api-access-ftnmr\") pod \"isvc-primary-fa781c-predictor-79f5fdb75f-27fcr\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") " pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.513199 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.513095    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:20.785830 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:20.785782    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"]
Oct 31 07:40:20.788527 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:40:20.788498    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032d76f7_5c87_4f22_b127_76d386e60b2c.slice/crio-45a0a50dd5975d1240cc8ddcce641edde7d3d57784b6b20da448e3281a946a3a WatchSource:0}: Error finding container 45a0a50dd5975d1240cc8ddcce641edde7d3d57784b6b20da448e3281a946a3a: Status 404 returned error can't find the container with id 45a0a50dd5975d1240cc8ddcce641edde7d3d57784b6b20da448e3281a946a3a
Oct 31 07:40:21.128641 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:21.128546    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" event={"ID":"032d76f7-5c87-4f22-b127-76d386e60b2c","Type":"ContainerStarted","Data":"293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a"}
Oct 31 07:40:21.128641 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:21.128610    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" event={"ID":"032d76f7-5c87-4f22-b127-76d386e60b2c","Type":"ContainerStarted","Data":"45a0a50dd5975d1240cc8ddcce641edde7d3d57784b6b20da448e3281a946a3a"}
Oct 31 07:40:22.757816 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:22.757772    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Oct 31 07:40:24.143595 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:24.143501    4913 generic.go:356] "Generic (PLEG): container finished" podID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerID="e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506" exitCode=0
Oct 31 07:40:24.143595 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:24.143558    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerDied","Data":"e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506"}
Oct 31 07:40:24.766076 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:24.766033    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:40:24.766451 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:24.766423    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:40:25.149424 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:25.149341    4913 generic.go:356] "Generic (PLEG): container finished" podID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerID="293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a" exitCode=0
Oct 31 07:40:25.149781 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:25.149411    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" event={"ID":"032d76f7-5c87-4f22-b127-76d386e60b2c","Type":"ContainerDied","Data":"293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a"}
Oct 31 07:40:26.156291 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:26.156246    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" event={"ID":"032d76f7-5c87-4f22-b127-76d386e60b2c","Type":"ContainerStarted","Data":"6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529"}
Oct 31 07:40:26.156838 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:26.156337    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" event={"ID":"032d76f7-5c87-4f22-b127-76d386e60b2c","Type":"ContainerStarted","Data":"d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e"}
Oct 31 07:40:26.156838 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:26.156642    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:26.175991 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:26.175947    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podStartSLOduration=2.157641682 podStartE2EDuration="6.175935296s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.018293614s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:40:26.174726713 +0000 UTC m=+674.864495016" watchObservedRunningTime="2026-10-31 07:40:26.175935296 +0000 UTC m=+674.865703588"
Oct 31 07:40:27.160338 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:27.160308    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:27.161584 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:27.161554    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:40:27.757786 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:27.757727    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Oct 31 07:40:28.164692 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:28.164608    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:40:32.758059 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:32.758015    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Oct 31 07:40:32.758623 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:32.758155    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:40:33.168863 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:33.168782    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:40:33.169295 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:33.169268    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:40:34.766327 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:34.766274    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:40:34.766742 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:34.766610    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:40:37.758173 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:37.758130    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Oct 31 07:40:42.758081 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:42.758038    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Oct 31 07:40:43.169591 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:43.169490    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:40:44.766657 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:44.766609    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Oct 31 07:40:44.767093 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:44.766759    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:40:44.767093 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:44.766988    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:40:44.767093 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:44.767091    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:40:47.758124 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:47.758086    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Oct 31 07:40:48.800860 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.800829    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:40:48.911599 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.911512    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-cabundle-cert\") pod \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") "
Oct 31 07:40:48.911599 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.911568    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d963654-fdde-4615-98f1-a8fdac7dfc9e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d963654-fdde-4615-98f1-a8fdac7dfc9e-proxy-tls\") pod \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") "
Oct 31 07:40:48.911853 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.911686    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-rbac-proxy-sar-config\") pod \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") "
Oct 31 07:40:48.911853 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.911737    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-api-access-88z2g\" (UniqueName: \"kubernetes.io/projected/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-api-access-88z2g\") pod \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") "
Oct 31 07:40:48.911853 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.911756    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kserve-provision-location\") pod \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\" (UID: \"9d963654-fdde-4615-98f1-a8fdac7dfc9e\") "
Oct 31 07:40:48.912019 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.911961    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-cabundle-cert" pod "9d963654-fdde-4615-98f1-a8fdac7dfc9e" (UID: "9d963654-fdde-4615-98f1-a8fdac7dfc9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:40:48.912114 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.912089    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-rbac-proxy-sar-config" pod "9d963654-fdde-4615-98f1-a8fdac7dfc9e" (UID: "9d963654-fdde-4615-98f1-a8fdac7dfc9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:40:48.912179 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.912134    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kserve-provision-location" pod "9d963654-fdde-4615-98f1-a8fdac7dfc9e" (UID: "9d963654-fdde-4615-98f1-a8fdac7dfc9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:40:48.913823 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.913800    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-api-access-88z2g" pod "9d963654-fdde-4615-98f1-a8fdac7dfc9e" (UID: "9d963654-fdde-4615-98f1-a8fdac7dfc9e"). InnerVolumeSpecName "kube-api-access-88z2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:40:48.913873 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:48.913826    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d963654-fdde-4615-98f1-a8fdac7dfc9e-proxy-tls" pod "9d963654-fdde-4615-98f1-a8fdac7dfc9e" (UID: "9d963654-fdde-4615-98f1-a8fdac7dfc9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:40:49.012517 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.012482    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:49.012517 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.012509    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d963654-fdde-4615-98f1-a8fdac7dfc9e-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:49.012517 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.012519    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:49.012748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.012529    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88z2g\" (UniqueName: \"kubernetes.io/projected/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kube-api-access-88z2g\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:49.012748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.012539    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d963654-fdde-4615-98f1-a8fdac7dfc9e-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:40:49.253885 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.253852    4913 generic.go:356] "Generic (PLEG): container finished" podID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerID="bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946" exitCode=0
Oct 31 07:40:49.254051 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.253914    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"
Oct 31 07:40:49.254051 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.253928    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerDied","Data":"bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946"}
Oct 31 07:40:49.254051 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.253990    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz" event={"ID":"9d963654-fdde-4615-98f1-a8fdac7dfc9e","Type":"ContainerDied","Data":"6c33ccc16e16d2ff7927cb1256bcaed25b64993af4451443510cc8588727d790"}
Oct 31 07:40:49.254051 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.254021    4913 scope.go:122] "RemoveContainer" containerID="bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946"
Oct 31 07:40:49.263062 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.263041    4913 scope.go:122] "RemoveContainer" containerID="030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7"
Oct 31 07:40:49.270344 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.270328    4913 scope.go:122] "RemoveContainer" containerID="e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506"
Oct 31 07:40:49.273878 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.273859    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"]
Oct 31 07:40:49.277952 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.277932    4913 scope.go:122] "RemoveContainer" containerID="ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835"
Oct 31 07:40:49.278281 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.278250    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-predictor-7468d78d5-bvkcz"]
Oct 31 07:40:49.284976 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.284962    4913 scope.go:122] "RemoveContainer" containerID="bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946"
Oct 31 07:40:49.285251 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:40:49.285231    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946\": container with ID starting with bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946 not found: ID does not exist" containerID="bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946"
Oct 31 07:40:49.285316 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.285260    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946"} err="failed to get container status \"bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946\": rpc error: code = NotFound desc = could not find container \"bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946\": container with ID starting with bf4252329afa087306627a1fd3f99d3400497b7902e515192734c381461db946 not found: ID does not exist"
Oct 31 07:40:49.285316 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.285279    4913 scope.go:122] "RemoveContainer" containerID="030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7"
Oct 31 07:40:49.285534 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:40:49.285517    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7\": container with ID starting with 030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7 not found: ID does not exist" containerID="030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7"
Oct 31 07:40:49.285580 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.285539    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7"} err="failed to get container status \"030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7\": rpc error: code = NotFound desc = could not find container \"030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7\": container with ID starting with 030fbd3ebf02ce4f54a04948f6b25c19ab8090d1cb607042429624f5eabbc8c7 not found: ID does not exist"
Oct 31 07:40:49.285580 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.285553    4913 scope.go:122] "RemoveContainer" containerID="e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506"
Oct 31 07:40:49.285793 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:40:49.285778    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506\": container with ID starting with e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506 not found: ID does not exist" containerID="e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506"
Oct 31 07:40:49.285844 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.285807    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506"} err="failed to get container status \"e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506\": rpc error: code = NotFound desc = could not find container \"e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506\": container with ID starting with e052ff576db59ba345b7deaa24f527ff236c7623c1ef440d7dab3bba735d8506 not found: ID does not exist"
Oct 31 07:40:49.285844 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.285819    4913 scope.go:122] "RemoveContainer" containerID="ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835"
Oct 31 07:40:49.286047 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:40:49.286031    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835\": container with ID starting with ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835 not found: ID does not exist" containerID="ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835"
Oct 31 07:40:49.286095 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.286050    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835"} err="failed to get container status \"ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835\": rpc error: code = NotFound desc = could not find container \"ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835\": container with ID starting with ec018a49e5716984ade761ed3a53963cc43ae59c6fb0dc67dc75063522cc7835 not found: ID does not exist"
Oct 31 07:40:49.399447 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:40:49.399410    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:40:49.418857 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:49.418824    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" path="/var/lib/kubelet/pods/9d963654-fdde-4615-98f1-a8fdac7dfc9e/volumes"
Oct 31 07:40:50.442359 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442314    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"]
Oct 31 07:40:50.442875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442797    4913 cpu_manager.go:414] "containerMap: removing container" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy"
Oct 31 07:40:50.442875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442815    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy"
Oct 31 07:40:50.442875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442828    4913 cpu_manager.go:414] "containerMap: removing container" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent"
Oct 31 07:40:50.442875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442836    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent"
Oct 31 07:40:50.442875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442863    4913 cpu_manager.go:414] "containerMap: removing container" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="storage-initializer"
Oct 31 07:40:50.442875 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442871    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="storage-initializer"
Oct 31 07:40:50.443240 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442889    4913 cpu_manager.go:414] "containerMap: removing container" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container"
Oct 31 07:40:50.443240 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.442897    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container"
Oct 31 07:40:50.447754 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.443933    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kserve-container"
Oct 31 07:40:50.447754 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.443978    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="kube-rbac-proxy"
Oct 31 07:40:50.447754 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.443989    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d963654-fdde-4615-98f1-a8fdac7dfc9e" containerName="agent"
Oct 31 07:40:50.451595 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.451568    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.458483 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.458452    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"]
Oct 31 07:40:50.460034 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.460008    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:40:50.460425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.460233    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:40:50.460425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.460336    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:40:50.460425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.460368    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:40:50.460425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.460368    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Oct 31 07:40:50.460619 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.460387    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Oct 31 07:40:50.460619 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.460407    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 07:40:50.525075 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.525031    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8612d908-a5c3-4c39-998c-c42f450cc3c3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.525274 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.525092    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.525274 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.525217    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.525274 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.525271    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8612d908-a5c3-4c39-998c-c42f450cc3c3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.525412 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.525342    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswkt\" (UniqueName: \"kubernetes.io/projected/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-api-access-fswkt\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.625929 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.625893    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8612d908-a5c3-4c39-998c-c42f450cc3c3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.625929 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.625934    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswkt\" (UniqueName: \"kubernetes.io/projected/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-api-access-fswkt\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.626132 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.625961    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8612d908-a5c3-4c39-998c-c42f450cc3c3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.626132 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.626076    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.626286 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.626132    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.626427 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.626401    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8612d908-a5c3-4c39-998c-c42f450cc3c3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.626801 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.626783    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.626920 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.626894    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.628535 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.628503    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8612d908-a5c3-4c39-998c-c42f450cc3c3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.632930 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.632908    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswkt\" (UniqueName: \"kubernetes.io/projected/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-api-access-fswkt\") pod \"isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") " pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:50.773743 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:50.773641    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:51.048584 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:51.048508    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"]
Oct 31 07:40:51.050824 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:40:51.050796    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8612d908_a5c3_4c39_998c_c42f450cc3c3.slice/crio-6acbcd3d7c423c4dc5b5a93d3db6608ef2a7f1e5660e561c6e7ca39d4eab10fa WatchSource:0}: Error finding container 6acbcd3d7c423c4dc5b5a93d3db6608ef2a7f1e5660e561c6e7ca39d4eab10fa: Status 404 returned error can't find the container with id 6acbcd3d7c423c4dc5b5a93d3db6608ef2a7f1e5660e561c6e7ca39d4eab10fa
Oct 31 07:40:51.265916 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:51.265870    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerStarted","Data":"d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952"}
Oct 31 07:40:51.266105 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:51.265937    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerStarted","Data":"6acbcd3d7c423c4dc5b5a93d3db6608ef2a7f1e5660e561c6e7ca39d4eab10fa"}
Oct 31 07:40:53.169741 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:53.169701    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:40:55.283782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:55.283746    4913 generic.go:356] "Generic (PLEG): container finished" podID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerID="d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952" exitCode=0
Oct 31 07:40:55.284202 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:55.283823    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerDied","Data":"d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952"}
Oct 31 07:40:56.291699 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.291661    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerStarted","Data":"480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c"}
Oct 31 07:40:56.292148 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.291754    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerStarted","Data":"de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87"}
Oct 31 07:40:56.292148 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.291777    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerStarted","Data":"9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d"}
Oct 31 07:40:56.292148 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.292086    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:56.292148 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.292141    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:56.292399 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.292165    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:40:56.293814 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.293782    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:40:56.294494 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.294474    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:40:56.313011 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:56.312957    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podStartSLOduration=2.246496566 podStartE2EDuration="6.312943241s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.066446675s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:40:56.31148652 +0000 UTC m=+705.001254832" watchObservedRunningTime="2026-10-31 07:40:56.312943241 +0000 UTC m=+705.002711534"
Oct 31 07:40:57.296749 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:57.296709    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:40:57.297297 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:40:57.296991    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:41:02.300849 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:02.300820    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:41:02.301533 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:02.301503    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:41:02.301766 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:02.301741    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:41:03.169301 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:03.169260    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:41:12.301542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:12.301487    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:41:12.301972 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:12.301891    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:41:13.169698 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:13.169661    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:41:19.399008 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:41:19.398970    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:41:22.302253 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:22.302214    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:41:22.302748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:22.302719    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:41:23.169881 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:23.169843    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:41:32.301815 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:32.301769    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:41:32.302363 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:32.302206    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:41:33.169674 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:33.169636    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:41:42.301707 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:42.301669    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:41:42.302125 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:42.302069    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:41:43.169418 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:43.169381    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Oct 31 07:41:49.399767 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:41:49.399723    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:41:52.301797 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:52.301757    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:41:52.302232 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:52.302214    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:41:53.170005 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:41:53.169922    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:42:02.301878 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:02.301821    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:42:02.302335 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:02.302228    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:42:10.414952 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:10.414909    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:42:10.415640 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:10.415259    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:42:15.650020 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.649980    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"]
Oct 31 07:42:15.654016 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.653990    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.658453 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.658427    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-secondary-fa781c-predictor-serving-cert\""
Oct 31 07:42:15.658591 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.658453    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-secondary-fa781c-kube-rbac-proxy-sar-config\""
Oct 31 07:42:15.658591 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.658453    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"invalid-s3-secret-fa781c\""
Oct 31 07:42:15.658591 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.658496    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"invalid-s3-sa-fa781c-dockercfg-42h2z\""
Oct 31 07:42:15.662975 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.662955    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"]
Oct 31 07:42:15.709549 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.709516    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwxx\" (UniqueName: \"kubernetes.io/projected/835de74f-95b1-48a8-9843-dc5660772564-kube-api-access-vnwxx\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.709695 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.709558    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835de74f-95b1-48a8-9843-dc5660772564-kserve-provision-location\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.709695 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.709582    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835de74f-95b1-48a8-9843-dc5660772564-proxy-tls\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.709773 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.709739    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.709773 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.709759    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-cabundle-cert\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.810923 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.810884    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwxx\" (UniqueName: \"kubernetes.io/projected/835de74f-95b1-48a8-9843-dc5660772564-kube-api-access-vnwxx\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.810923 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.810934    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835de74f-95b1-48a8-9843-dc5660772564-kserve-provision-location\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.811218 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.811051    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835de74f-95b1-48a8-9843-dc5660772564-proxy-tls\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.811218 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.811161    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.811218 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.811210    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-cabundle-cert\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.811465 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.811441    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835de74f-95b1-48a8-9843-dc5660772564-kserve-provision-location\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.811846 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.811821    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.811949 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.811885    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-cabundle-cert\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.813457 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.813435    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835de74f-95b1-48a8-9843-dc5660772564-proxy-tls\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.818509 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.818486    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwxx\" (UniqueName: \"kubernetes.io/projected/835de74f-95b1-48a8-9843-dc5660772564-kube-api-access-vnwxx\") pod \"isvc-secondary-fa781c-predictor-79d68c569b-qv5vw\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") " pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:15.972751 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:15.972719    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:16.244816 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:16.244790    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"]
Oct 31 07:42:16.247661 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:42:16.247621    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835de74f_95b1_48a8_9843_dc5660772564.slice/crio-f148c33f1b416e420d1672c795ca9c0c9b1378474aeda6c5ba888d3dde7b2d32 WatchSource:0}: Error finding container f148c33f1b416e420d1672c795ca9c0c9b1378474aeda6c5ba888d3dde7b2d32: Status 404 returned error can't find the container with id f148c33f1b416e420d1672c795ca9c0c9b1378474aeda6c5ba888d3dde7b2d32
Oct 31 07:42:16.624278 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:16.624160    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw" event={"ID":"835de74f-95b1-48a8-9843-dc5660772564","Type":"ContainerStarted","Data":"967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c"}
Oct 31 07:42:16.624278 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:16.624249    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw" event={"ID":"835de74f-95b1-48a8-9843-dc5660772564","Type":"ContainerStarted","Data":"f148c33f1b416e420d1672c795ca9c0c9b1378474aeda6c5ba888d3dde7b2d32"}
Oct 31 07:42:19.399969 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:19.399932    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:42:20.415381 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:20.415350    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:42:20.415774 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:20.415705    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:42:23.654385 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:23.654355    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_835de74f-95b1-48a8-9843-dc5660772564/storage-initializer/0.log"
Oct 31 07:42:23.654783 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:23.654394    4913 generic.go:356] "Generic (PLEG): container finished" podID="835de74f-95b1-48a8-9843-dc5660772564" containerID="967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c" exitCode=1
Oct 31 07:42:23.654783 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:23.654471    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw" event={"ID":"835de74f-95b1-48a8-9843-dc5660772564","Type":"ContainerDied","Data":"967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c"}
Oct 31 07:42:24.660477 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:24.660449    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_835de74f-95b1-48a8-9843-dc5660772564/storage-initializer/0.log"
Oct 31 07:42:24.660859 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:24.660528    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw" event={"ID":"835de74f-95b1-48a8-9843-dc5660772564","Type":"ContainerStarted","Data":"6ab663d38e4ad435155fe4b8bfffa4d09aa5d53621394adc63995b573542a09e"}
Oct 31 07:42:27.675759 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:27.675729    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_835de74f-95b1-48a8-9843-dc5660772564/storage-initializer/1.log"
Oct 31 07:42:27.676159 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:27.676143    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_835de74f-95b1-48a8-9843-dc5660772564/storage-initializer/0.log"
Oct 31 07:42:27.676238 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:27.676205    4913 generic.go:356] "Generic (PLEG): container finished" podID="835de74f-95b1-48a8-9843-dc5660772564" containerID="6ab663d38e4ad435155fe4b8bfffa4d09aa5d53621394adc63995b573542a09e" exitCode=1
Oct 31 07:42:27.676283 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:27.676247    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw" event={"ID":"835de74f-95b1-48a8-9843-dc5660772564","Type":"ContainerDied","Data":"6ab663d38e4ad435155fe4b8bfffa4d09aa5d53621394adc63995b573542a09e"}
Oct 31 07:42:27.676334 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:27.676307    4913 scope.go:122] "RemoveContainer" containerID="967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c"
Oct 31 07:42:27.676689 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:27.676671    4913 scope.go:122] "RemoveContainer" containerID="967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c"
Oct 31 07:42:27.687401 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:27.687376    4913 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_e2e-gw1-9226a10e02(835de74f-95b1-48a8-9843-dc5660772564)\"" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw" podUID="835de74f-95b1-48a8-9843-dc5660772564"
Oct 31 07:42:27.687604 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:27.687577    4913 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_e2e-gw1-9226a10e02_835de74f-95b1-48a8-9843-dc5660772564_0 in pod sandbox f148c33f1b416e420d1672c795ca9c0c9b1378474aeda6c5ba888d3dde7b2d32 from index: no such id: '967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c'" containerID="967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c"
Oct 31 07:42:27.687683 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:27.687618    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_e2e-gw1-9226a10e02_835de74f-95b1-48a8-9843-dc5660772564_0 in pod sandbox f148c33f1b416e420d1672c795ca9c0c9b1378474aeda6c5ba888d3dde7b2d32 from index: no such id: '967939c52efc2900db8748f6bacd14a9a88b99236b8a3f7ea1266312ab15ab2c'"
Oct 31 07:42:28.681553 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:28.681527    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_835de74f-95b1-48a8-9843-dc5660772564/storage-initializer/1.log"
Oct 31 07:42:35.766863 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:35.766827    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"]
Oct 31 07:42:35.798500 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:35.798466    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"]
Oct 31 07:42:35.798783 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:35.798756    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container" containerID="cri-o://d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e" gracePeriod=30
Oct 31 07:42:35.798956 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:35.798937    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kube-rbac-proxy" containerID="cri-o://6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529" gracePeriod=30
Oct 31 07:42:35.919974 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:35.919954    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_835de74f-95b1-48a8-9843-dc5660772564/storage-initializer/1.log"
Oct 31 07:42:35.922169 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:35.922148    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:36.087337 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.087259    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835de74f-95b1-48a8-9843-dc5660772564-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835de74f-95b1-48a8-9843-dc5660772564-kserve-provision-location\") pod \"835de74f-95b1-48a8-9843-dc5660772564\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") "
Oct 31 07:42:36.087337 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.087311    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-kube-rbac-proxy-sar-config\") pod \"835de74f-95b1-48a8-9843-dc5660772564\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") "
Oct 31 07:42:36.087523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.087380    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-cabundle-cert\") pod \"835de74f-95b1-48a8-9843-dc5660772564\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") "
Oct 31 07:42:36.087523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.087405    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835de74f-95b1-48a8-9843-dc5660772564-proxy-tls\" (UniqueName: \"kubernetes.io/secret/835de74f-95b1-48a8-9843-dc5660772564-proxy-tls\") pod \"835de74f-95b1-48a8-9843-dc5660772564\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") "
Oct 31 07:42:36.087523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.087441    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835de74f-95b1-48a8-9843-dc5660772564-kube-api-access-vnwxx\" (UniqueName: \"kubernetes.io/projected/835de74f-95b1-48a8-9843-dc5660772564-kube-api-access-vnwxx\") pod \"835de74f-95b1-48a8-9843-dc5660772564\" (UID: \"835de74f-95b1-48a8-9843-dc5660772564\") "
Oct 31 07:42:36.087660 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.087618    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835de74f-95b1-48a8-9843-dc5660772564-kserve-provision-location" pod "835de74f-95b1-48a8-9843-dc5660772564" (UID: "835de74f-95b1-48a8-9843-dc5660772564"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:42:36.087761 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.087733    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-cabundle-cert" pod "835de74f-95b1-48a8-9843-dc5660772564" (UID: "835de74f-95b1-48a8-9843-dc5660772564"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:36.087824 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.087743    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-kube-rbac-proxy-sar-config" pod "835de74f-95b1-48a8-9843-dc5660772564" (UID: "835de74f-95b1-48a8-9843-dc5660772564"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:36.089674 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.089646    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835de74f-95b1-48a8-9843-dc5660772564-proxy-tls" pod "835de74f-95b1-48a8-9843-dc5660772564" (UID: "835de74f-95b1-48a8-9843-dc5660772564"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:36.089674 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.089656    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835de74f-95b1-48a8-9843-dc5660772564-kube-api-access-vnwxx" pod "835de74f-95b1-48a8-9843-dc5660772564" (UID: "835de74f-95b1-48a8-9843-dc5660772564"). InnerVolumeSpecName "kube-api-access-vnwxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:36.188257 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.188220    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:36.188257 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.188251    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/835de74f-95b1-48a8-9843-dc5660772564-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:36.188257 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.188262    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835de74f-95b1-48a8-9843-dc5660772564-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:36.188482 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.188272    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnwxx\" (UniqueName: \"kubernetes.io/projected/835de74f-95b1-48a8-9843-dc5660772564-kube-api-access-vnwxx\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:36.188482 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.188281    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/835de74f-95b1-48a8-9843-dc5660772564-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:36.268637 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.268605    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"]
Oct 31 07:42:36.269228 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.268912    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" containerID="cri-o://9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d" gracePeriod=30
Oct 31 07:42:36.269228 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.268960    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy" containerID="cri-o://de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87" gracePeriod=30
Oct 31 07:42:36.269228 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.268952    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" containerID="cri-o://480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c" gracePeriod=30
Oct 31 07:42:36.719580 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.719546    4913 generic.go:356] "Generic (PLEG): container finished" podID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerID="de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87" exitCode=0
Oct 31 07:42:36.719747 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.719619    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerDied","Data":"de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87"}
Oct 31 07:42:36.720906 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.720887    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-secondary-fa781c-predictor-79d68c569b-qv5vw_835de74f-95b1-48a8-9843-dc5660772564/storage-initializer/1.log"
Oct 31 07:42:36.721039 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.720960    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw" event={"ID":"835de74f-95b1-48a8-9843-dc5660772564","Type":"ContainerDied","Data":"f148c33f1b416e420d1672c795ca9c0c9b1378474aeda6c5ba888d3dde7b2d32"}
Oct 31 07:42:36.721039 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.720996    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"
Oct 31 07:42:36.721139 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.721033    4913 scope.go:122] "RemoveContainer" containerID="6ab663d38e4ad435155fe4b8bfffa4d09aa5d53621394adc63995b573542a09e"
Oct 31 07:42:36.723551 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.723525    4913 generic.go:356] "Generic (PLEG): container finished" podID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerID="6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529" exitCode=0
Oct 31 07:42:36.723690 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.723557    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" event={"ID":"032d76f7-5c87-4f22-b127-76d386e60b2c","Type":"ContainerDied","Data":"6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529"}
Oct 31 07:42:36.758337 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.758306    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"]
Oct 31 07:42:36.761641 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:36.761618    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-secondary-fa781c-predictor-79d68c569b-qv5vw"]
Oct 31 07:42:37.297798 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:37.297753    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Oct 31 07:42:37.419089 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:37.419055    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835de74f-95b1-48a8-9843-dc5660772564" path="/var/lib/kubelet/pods/835de74f-95b1-48a8-9843-dc5660772564/volumes"
Oct 31 07:42:38.164957 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:38.164917    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Oct 31 07:42:40.414825 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:40.414773    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:42:40.415299 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:40.415066    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:42:40.952996 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:40.952969    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:42:41.133025 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.132939    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-api-access-ftnmr\" (UniqueName: \"kubernetes.io/projected/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-api-access-ftnmr\") pod \"032d76f7-5c87-4f22-b127-76d386e60b2c\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") "
Oct 31 07:42:41.133172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.133019    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032d76f7-5c87-4f22-b127-76d386e60b2c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032d76f7-5c87-4f22-b127-76d386e60b2c-kserve-provision-location\") pod \"032d76f7-5c87-4f22-b127-76d386e60b2c\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") "
Oct 31 07:42:41.133172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.133125    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-cabundle-cert\") pod \"032d76f7-5c87-4f22-b127-76d386e60b2c\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") "
Oct 31 07:42:41.133172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.133167    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/032d76f7-5c87-4f22-b127-76d386e60b2c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/032d76f7-5c87-4f22-b127-76d386e60b2c-proxy-tls\") pod \"032d76f7-5c87-4f22-b127-76d386e60b2c\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") "
Oct 31 07:42:41.133364 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.133221    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-rbac-proxy-sar-config\") pod \"032d76f7-5c87-4f22-b127-76d386e60b2c\" (UID: \"032d76f7-5c87-4f22-b127-76d386e60b2c\") "
Oct 31 07:42:41.133364 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.133349    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032d76f7-5c87-4f22-b127-76d386e60b2c-kserve-provision-location" pod "032d76f7-5c87-4f22-b127-76d386e60b2c" (UID: "032d76f7-5c87-4f22-b127-76d386e60b2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:42:41.133524 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.133506    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032d76f7-5c87-4f22-b127-76d386e60b2c-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:41.133624 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.133602    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-cabundle-cert" pod "032d76f7-5c87-4f22-b127-76d386e60b2c" (UID: "032d76f7-5c87-4f22-b127-76d386e60b2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:41.133687 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.133666    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-rbac-proxy-sar-config" pod "032d76f7-5c87-4f22-b127-76d386e60b2c" (UID: "032d76f7-5c87-4f22-b127-76d386e60b2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:41.135223 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.135180    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-api-access-ftnmr" pod "032d76f7-5c87-4f22-b127-76d386e60b2c" (UID: "032d76f7-5c87-4f22-b127-76d386e60b2c"). InnerVolumeSpecName "kube-api-access-ftnmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:41.135223 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.135202    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/032d76f7-5c87-4f22-b127-76d386e60b2c-proxy-tls" pod "032d76f7-5c87-4f22-b127-76d386e60b2c" (UID: "032d76f7-5c87-4f22-b127-76d386e60b2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:41.234053 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.234028    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnmr\" (UniqueName: \"kubernetes.io/projected/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-api-access-ftnmr\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:41.234158 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.234056    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:41.234158 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.234067    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/032d76f7-5c87-4f22-b127-76d386e60b2c-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:41.234158 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.234075    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/032d76f7-5c87-4f22-b127-76d386e60b2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:41.749308 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.749276    4913 generic.go:356] "Generic (PLEG): container finished" podID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerID="d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e" exitCode=0
Oct 31 07:42:41.749808 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.749334    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"
Oct 31 07:42:41.749808 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.749357    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" event={"ID":"032d76f7-5c87-4f22-b127-76d386e60b2c","Type":"ContainerDied","Data":"d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e"}
Oct 31 07:42:41.749808 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.749411    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr" event={"ID":"032d76f7-5c87-4f22-b127-76d386e60b2c","Type":"ContainerDied","Data":"45a0a50dd5975d1240cc8ddcce641edde7d3d57784b6b20da448e3281a946a3a"}
Oct 31 07:42:41.749808 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.749443    4913 scope.go:122] "RemoveContainer" containerID="6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529"
Oct 31 07:42:41.752312 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.752289    4913 generic.go:356] "Generic (PLEG): container finished" podID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerID="9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d" exitCode=0
Oct 31 07:42:41.752434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.752329    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerDied","Data":"9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d"}
Oct 31 07:42:41.757988 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.757974    4913 scope.go:122] "RemoveContainer" containerID="d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e"
Oct 31 07:42:41.765725 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.765700    4913 scope.go:122] "RemoveContainer" containerID="293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a"
Oct 31 07:42:41.766996 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.766975    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"]
Oct 31 07:42:41.770648 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.770627    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-primary-fa781c-predictor-79f5fdb75f-27fcr"]
Oct 31 07:42:41.773844 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.773821    4913 scope.go:122] "RemoveContainer" containerID="6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529"
Oct 31 07:42:41.774134 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:41.774114    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529\": container with ID starting with 6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529 not found: ID does not exist" containerID="6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529"
Oct 31 07:42:41.774230 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.774149    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529"} err="failed to get container status \"6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529\": rpc error: code = NotFound desc = could not find container \"6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529\": container with ID starting with 6de8d32aa6e13061f05d3cdd582516d0d19117ad210ee2cc6bffc55e71524529 not found: ID does not exist"
Oct 31 07:42:41.774230 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.774173    4913 scope.go:122] "RemoveContainer" containerID="d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e"
Oct 31 07:42:41.774475 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:41.774457    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e\": container with ID starting with d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e not found: ID does not exist" containerID="d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e"
Oct 31 07:42:41.774530 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.774482    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e"} err="failed to get container status \"d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e\": rpc error: code = NotFound desc = could not find container \"d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e\": container with ID starting with d211369346a0ce8c0ec31865e7551202b80f24b83f4afc3fa9eac40a0324393e not found: ID does not exist"
Oct 31 07:42:41.774530 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.774500    4913 scope.go:122] "RemoveContainer" containerID="293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a"
Oct 31 07:42:41.774735 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:41.774716    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a\": container with ID starting with 293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a not found: ID does not exist" containerID="293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a"
Oct 31 07:42:41.774792 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:41.774741    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a"} err="failed to get container status \"293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a\": rpc error: code = NotFound desc = could not find container \"293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a\": container with ID starting with 293c083c3b2ae4aaeb5ad39b9da860b0365fd68bf54937b4ba10eda51b4cd16a not found: ID does not exist"
Oct 31 07:42:42.297432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.297389    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Oct 31 07:42:42.445432 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445399    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"]
Oct 31 07:42:42.445750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445736    4913 cpu_manager.go:414] "containerMap: removing container" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="storage-initializer"
Oct 31 07:42:42.445791 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445751    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="storage-initializer"
Oct 31 07:42:42.445791 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445774    4913 cpu_manager.go:414] "containerMap: removing container" podUID="835de74f-95b1-48a8-9843-dc5660772564" containerName="storage-initializer"
Oct 31 07:42:42.445791 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445779    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835de74f-95b1-48a8-9843-dc5660772564" containerName="storage-initializer"
Oct 31 07:42:42.445884 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445793    4913 cpu_manager.go:414] "containerMap: removing container" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container"
Oct 31 07:42:42.445884 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445798    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container"
Oct 31 07:42:42.445884 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445805    4913 cpu_manager.go:414] "containerMap: removing container" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kube-rbac-proxy"
Oct 31 07:42:42.445884 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445810    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kube-rbac-proxy"
Oct 31 07:42:42.445884 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445823    4913 cpu_manager.go:414] "containerMap: removing container" podUID="835de74f-95b1-48a8-9843-dc5660772564" containerName="storage-initializer"
Oct 31 07:42:42.445884 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445828    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835de74f-95b1-48a8-9843-dc5660772564" containerName="storage-initializer"
Oct 31 07:42:42.445884 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445877    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="835de74f-95b1-48a8-9843-dc5660772564" containerName="storage-initializer"
Oct 31 07:42:42.445884 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445885    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kserve-container"
Oct 31 07:42:42.446239 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.445891    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" containerName="kube-rbac-proxy"
Oct 31 07:42:42.446239 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.446000    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="835de74f-95b1-48a8-9843-dc5660772564" containerName="storage-initializer"
Oct 31 07:42:42.450758 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.450739    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.456208 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.456165    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"fail-s3-secret-b7aff4\""
Oct 31 07:42:42.457932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.457908    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"]
Oct 31 07:42:42.459172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.459129    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"fail-s3-sa-b7aff4-dockercfg-gvskr\""
Oct 31 07:42:42.459579 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.459536    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-init-fail-b7aff4-kube-rbac-proxy-sar-config\""
Oct 31 07:42:42.459688 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.459584    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:42:42.460335 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.460312    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:42:42.460335 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.460317    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:42:42.500260 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.500232    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-init-fail-b7aff4-predictor-serving-cert\""
Oct 31 07:42:42.544251 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.544212    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-proxy-tls\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.544399 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.544261    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-cabundle-cert\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.544399 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.544331    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.544488 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.544431    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kserve-provision-location\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.544523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.544490    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k557s\" (UniqueName: \"kubernetes.io/projected/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-api-access-k557s\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.645031 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.644953    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-proxy-tls\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.645031 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.644994    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-cabundle-cert\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.645031 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.645022    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.645323 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.645052    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kserve-provision-location\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.645323 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.645081    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k557s\" (UniqueName: \"kubernetes.io/projected/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-api-access-k557s\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.645637 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.645604    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kserve-provision-location\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.645810 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.645786    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-cabundle-cert\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.645873 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.645830    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.647518 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.647496    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-proxy-tls\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.654199 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.654167    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k557s\" (UniqueName: \"kubernetes.io/projected/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-api-access-k557s\") pod \"isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") " pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:42.769416 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:42.769381    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:43.041922 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:43.041899    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"]
Oct 31 07:42:43.044740 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:42:43.044696    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d80f3f_4905_4f88_a030_5c7bf70e1f4e.slice/crio-93bcb98afbdacc7ec63a070ad0d01d269755022c4c683c7754aa8091d278b19f WatchSource:0}: Error finding container 93bcb98afbdacc7ec63a070ad0d01d269755022c4c683c7754aa8091d278b19f: Status 404 returned error can't find the container with id 93bcb98afbdacc7ec63a070ad0d01d269755022c4c683c7754aa8091d278b19f
Oct 31 07:42:43.419941 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:43.419859    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032d76f7-5c87-4f22-b127-76d386e60b2c" path="/var/lib/kubelet/pods/032d76f7-5c87-4f22-b127-76d386e60b2c/volumes"
Oct 31 07:42:43.764708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:43.764675    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4" event={"ID":"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e","Type":"ContainerStarted","Data":"07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a"}
Oct 31 07:42:43.764880 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:43.764732    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4" event={"ID":"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e","Type":"ContainerStarted","Data":"93bcb98afbdacc7ec63a070ad0d01d269755022c4c683c7754aa8091d278b19f"}
Oct 31 07:42:47.297634 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:47.297588    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Oct 31 07:42:47.298033 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:47.297722    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:42:47.783688 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:47.783662    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e/storage-initializer/0.log"
Oct 31 07:42:47.783840 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:47.783699    4913 generic.go:356] "Generic (PLEG): container finished" podID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" containerID="07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a" exitCode=1
Oct 31 07:42:47.783840 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:47.783764    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4" event={"ID":"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e","Type":"ContainerDied","Data":"07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a"}
Oct 31 07:42:48.790715 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:48.790687    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e/storage-initializer/0.log"
Oct 31 07:42:48.791103 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:48.790772    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4" event={"ID":"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e","Type":"ContainerStarted","Data":"a9aa7f9bf16bd2e9b00c96606f32046e1d24427d3035cfec14296c4199bffa4d"}
Oct 31 07:42:49.399893 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:49.399857    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:42:50.415380 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:50.415327    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:42:50.415771 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:50.415660    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:42:51.804585 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:51.804562    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e/storage-initializer/1.log"
Oct 31 07:42:51.805010 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:51.804958    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e/storage-initializer/0.log"
Oct 31 07:42:51.805010 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:51.804997    4913 generic.go:356] "Generic (PLEG): container finished" podID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" containerID="a9aa7f9bf16bd2e9b00c96606f32046e1d24427d3035cfec14296c4199bffa4d" exitCode=1
Oct 31 07:42:51.805107 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:51.805062    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4" event={"ID":"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e","Type":"ContainerDied","Data":"a9aa7f9bf16bd2e9b00c96606f32046e1d24427d3035cfec14296c4199bffa4d"}
Oct 31 07:42:51.805157 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:51.805120    4913 scope.go:122] "RemoveContainer" containerID="07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a"
Oct 31 07:42:51.805489 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:51.805472    4913 scope.go:122] "RemoveContainer" containerID="07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a"
Oct 31 07:42:51.815577 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:51.815555    4913 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_e2e-gw1-9226a10e02(a0d80f3f-4905-4f88-a030-5c7bf70e1f4e)\"" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4" podUID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e"
Oct 31 07:42:51.815820 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:42:51.815794    4913 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_e2e-gw1-9226a10e02_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e_0 in pod sandbox 93bcb98afbdacc7ec63a070ad0d01d269755022c4c683c7754aa8091d278b19f from index: no such id: '07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a'" containerID="07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a"
Oct 31 07:42:51.815892 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:51.815828    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_e2e-gw1-9226a10e02_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e_0 in pod sandbox 93bcb98afbdacc7ec63a070ad0d01d269755022c4c683c7754aa8091d278b19f from index: no such id: '07fe10788dcb5f7616fd49ce11752e19cba6236b56401f31658fd9dd8e83af7a'"
Oct 31 07:42:52.296888 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:52.296847    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Oct 31 07:42:52.504531 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:52.504497    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"]
Oct 31 07:42:52.810992 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:52.810965    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e/storage-initializer/1.log"
Oct 31 07:42:52.933241 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:52.933219    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e/storage-initializer/1.log"
Oct 31 07:42:52.937260 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:52.937242    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:53.030850 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.030815    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-proxy-tls\") pod \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") "
Oct 31 07:42:53.031012 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.030858    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-cabundle-cert\") pod \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") "
Oct 31 07:42:53.031012 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.030885    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-rbac-proxy-sar-config\") pod \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") "
Oct 31 07:42:53.031012 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.030959    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-api-access-k557s\" (UniqueName: \"kubernetes.io/projected/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-api-access-k557s\") pod \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") "
Oct 31 07:42:53.031012 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.030995    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kserve-provision-location\") pod \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\" (UID: \"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e\") "
Oct 31 07:42:53.031353 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.031324    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kserve-provision-location" pod "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" (UID: "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:42:53.031353 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.031273    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-cabundle-cert" pod "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" (UID: "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:53.031474 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.031331    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-rbac-proxy-sar-config" pod "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" (UID: "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:53.033060 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.033041    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-api-access-k557s" pod "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" (UID: "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e"). InnerVolumeSpecName "kube-api-access-k557s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:53.033110 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.033066    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-proxy-tls" pod "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" (UID: "a0d80f3f-4905-4f88-a030-5c7bf70e1f4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:53.131931 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.131855    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k557s\" (UniqueName: \"kubernetes.io/projected/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-api-access-k557s\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:53.131931 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.131883    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:53.131931 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.131893    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:53.131931 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.131905    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:53.131931 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.131914    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:42:53.816367 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.816292    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4_a0d80f3f-4905-4f88-a030-5c7bf70e1f4e/storage-initializer/1.log"
Oct 31 07:42:53.816759 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.816389    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"
Oct 31 07:42:53.816759 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.816426    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4" event={"ID":"a0d80f3f-4905-4f88-a030-5c7bf70e1f4e","Type":"ContainerDied","Data":"93bcb98afbdacc7ec63a070ad0d01d269755022c4c683c7754aa8091d278b19f"}
Oct 31 07:42:53.816759 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.816499    4913 scope.go:122] "RemoveContainer" containerID="a9aa7f9bf16bd2e9b00c96606f32046e1d24427d3035cfec14296c4199bffa4d"
Oct 31 07:42:53.845997 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.845966    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"]
Oct 31 07:42:53.851577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:53.851549    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-init-fail-b7aff4-predictor-7d88849c66-s8jz4"]
Oct 31 07:42:55.418761 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:55.418718    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" path="/var/lib/kubelet/pods/a0d80f3f-4905-4f88-a030-5c7bf70e1f4e/volumes"
Oct 31 07:42:57.296861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:42:57.296814    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Oct 31 07:43:00.414819 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:00.414778    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:5000: connect: connection refused"
Oct 31 07:43:00.415260 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:00.414933    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:43:00.415260 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:00.415019    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:43:00.415260 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:00.415142    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:43:02.297213 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:02.297137    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Oct 31 07:43:06.422721 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.422697    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:43:06.533846 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.533765    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-api-access-fswkt\" (UniqueName: \"kubernetes.io/projected/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-api-access-fswkt\") pod \"8612d908-a5c3-4c39-998c-c42f450cc3c3\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") "
Oct 31 07:43:06.533846 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.533827    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8612d908-a5c3-4c39-998c-c42f450cc3c3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8612d908-a5c3-4c39-998c-c42f450cc3c3-kserve-provision-location\") pod \"8612d908-a5c3-4c39-998c-c42f450cc3c3\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") "
Oct 31 07:43:06.534088 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.533855    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-rbac-proxy-sar-config\") pod \"8612d908-a5c3-4c39-998c-c42f450cc3c3\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") "
Oct 31 07:43:06.534088 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.533928    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8612d908-a5c3-4c39-998c-c42f450cc3c3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8612d908-a5c3-4c39-998c-c42f450cc3c3-proxy-tls\") pod \"8612d908-a5c3-4c39-998c-c42f450cc3c3\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") "
Oct 31 07:43:06.534088 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.533981    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-cabundle-cert\") pod \"8612d908-a5c3-4c39-998c-c42f450cc3c3\" (UID: \"8612d908-a5c3-4c39-998c-c42f450cc3c3\") "
Oct 31 07:43:06.534275 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.534157    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8612d908-a5c3-4c39-998c-c42f450cc3c3-kserve-provision-location" pod "8612d908-a5c3-4c39-998c-c42f450cc3c3" (UID: "8612d908-a5c3-4c39-998c-c42f450cc3c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:43:06.534322 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.534206    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-rbac-proxy-sar-config" pod "8612d908-a5c3-4c39-998c-c42f450cc3c3" (UID: "8612d908-a5c3-4c39-998c-c42f450cc3c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:43:06.534381 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.534363    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-cabundle-cert" pod "8612d908-a5c3-4c39-998c-c42f450cc3c3" (UID: "8612d908-a5c3-4c39-998c-c42f450cc3c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:43:06.535984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.535961    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8612d908-a5c3-4c39-998c-c42f450cc3c3-proxy-tls" pod "8612d908-a5c3-4c39-998c-c42f450cc3c3" (UID: "8612d908-a5c3-4c39-998c-c42f450cc3c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:43:06.536033 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.536005    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-api-access-fswkt" pod "8612d908-a5c3-4c39-998c-c42f450cc3c3" (UID: "8612d908-a5c3-4c39-998c-c42f450cc3c3"). InnerVolumeSpecName "kube-api-access-fswkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:43:06.634571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.634525    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8612d908-a5c3-4c39-998c-c42f450cc3c3-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:43:06.634571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.634565    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:43:06.634571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.634575    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8612d908-a5c3-4c39-998c-c42f450cc3c3-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:43:06.634571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.634585    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8612d908-a5c3-4c39-998c-c42f450cc3c3-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:43:06.634835 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.634594    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fswkt\" (UniqueName: \"kubernetes.io/projected/8612d908-a5c3-4c39-998c-c42f450cc3c3-kube-api-access-fswkt\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:43:06.873665 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.873578    4913 generic.go:356] "Generic (PLEG): container finished" podID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerID="480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c" exitCode=0
Oct 31 07:43:06.873665 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.873639    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"
Oct 31 07:43:06.873665 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.873654    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerDied","Data":"480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c"}
Oct 31 07:43:06.873901 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.873708    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll" event={"ID":"8612d908-a5c3-4c39-998c-c42f450cc3c3","Type":"ContainerDied","Data":"6acbcd3d7c423c4dc5b5a93d3db6608ef2a7f1e5660e561c6e7ca39d4eab10fa"}
Oct 31 07:43:06.873901 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.873740    4913 scope.go:122] "RemoveContainer" containerID="480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c"
Oct 31 07:43:06.882167 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.882141    4913 scope.go:122] "RemoveContainer" containerID="de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87"
Oct 31 07:43:06.889704 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.889690    4913 scope.go:122] "RemoveContainer" containerID="9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d"
Oct 31 07:43:06.894950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.894925    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"]
Oct 31 07:43:06.897137 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.897115    4913 scope.go:122] "RemoveContainer" containerID="d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952"
Oct 31 07:43:06.898889 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.898872    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-sklearn-batcher-custom-predictor-76c6d878c5-rklll"]
Oct 31 07:43:06.904851 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.904821    4913 scope.go:122] "RemoveContainer" containerID="480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c"
Oct 31 07:43:06.905109 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:43:06.905092    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c\": container with ID starting with 480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c not found: ID does not exist" containerID="480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c"
Oct 31 07:43:06.905154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.905118    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c"} err="failed to get container status \"480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c\": rpc error: code = NotFound desc = could not find container \"480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c\": container with ID starting with 480b7af9fcca40884360abf2171d08568f3aa6060e445175b67df0e4d1c1693c not found: ID does not exist"
Oct 31 07:43:06.905154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.905135    4913 scope.go:122] "RemoveContainer" containerID="de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87"
Oct 31 07:43:06.905383 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:43:06.905367    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87\": container with ID starting with de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87 not found: ID does not exist" containerID="de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87"
Oct 31 07:43:06.905425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.905387    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87"} err="failed to get container status \"de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87\": rpc error: code = NotFound desc = could not find container \"de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87\": container with ID starting with de2990ca6f1bd0d61bd5534eda66bb85cacdf34c72639857dda89883f06fed87 not found: ID does not exist"
Oct 31 07:43:06.905425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.905398    4913 scope.go:122] "RemoveContainer" containerID="9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d"
Oct 31 07:43:06.905591 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:43:06.905577    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d\": container with ID starting with 9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d not found: ID does not exist" containerID="9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d"
Oct 31 07:43:06.905627 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.905593    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d"} err="failed to get container status \"9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d\": rpc error: code = NotFound desc = could not find container \"9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d\": container with ID starting with 9490964d0da34cbc101be311b3c6b131e78155ffe9793fac9a10e85444014d8d not found: ID does not exist"
Oct 31 07:43:06.905627 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.905603    4913 scope.go:122] "RemoveContainer" containerID="d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952"
Oct 31 07:43:06.905817 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:43:06.905798    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952\": container with ID starting with d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952 not found: ID does not exist" containerID="d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952"
Oct 31 07:43:06.905854 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:06.905826    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952"} err="failed to get container status \"d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952\": rpc error: code = NotFound desc = could not find container \"d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952\": container with ID starting with d7ab6d493431bbc4ca5d717bf07e168d76143ed55b505a311f5fdf6fa62fc952 not found: ID does not exist"
Oct 31 07:43:07.419660 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:07.419628    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" path="/var/lib/kubelet/pods/8612d908-a5c3-4c39-998c-c42f450cc3c3/volumes"
Oct 31 07:43:08.053885 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.053848    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"]
Oct 31 07:43:08.054365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054348    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container"
Oct 31 07:43:08.054409 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054368    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container"
Oct 31 07:43:08.054409 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054392    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy"
Oct 31 07:43:08.054409 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054399    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy"
Oct 31 07:43:08.054501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054413    4913 cpu_manager.go:414] "containerMap: removing container" podUID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" containerName="storage-initializer"
Oct 31 07:43:08.054501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054420    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" containerName="storage-initializer"
Oct 31 07:43:08.054501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054444    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent"
Oct 31 07:43:08.054501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054451    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent"
Oct 31 07:43:08.054501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054460    4913 cpu_manager.go:414] "containerMap: removing container" podUID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" containerName="storage-initializer"
Oct 31 07:43:08.054501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054466    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" containerName="storage-initializer"
Oct 31 07:43:08.054501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054486    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="storage-initializer"
Oct 31 07:43:08.054501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054492    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="storage-initializer"
Oct 31 07:43:08.054720 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054557    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kserve-container"
Oct 31 07:43:08.054720 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054570    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="kube-rbac-proxy"
Oct 31 07:43:08.054720 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054580    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" containerName="storage-initializer"
Oct 31 07:43:08.054720 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054591    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d80f3f-4905-4f88-a030-5c7bf70e1f4e" containerName="storage-initializer"
Oct 31 07:43:08.054720 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.054598    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8612d908-a5c3-4c39-998c-c42f450cc3c3" containerName="agent"
Oct 31 07:43:08.059495 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.059467    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:43:08.069717 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.069668    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:43:08.069894 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.069805    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:43:08.070034 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.069917    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:43:08.071234 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.071208    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"]
Oct 31 07:43:08.144812 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.144781    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jsb\" (UniqueName: \"kubernetes.io/projected/46fcdc88-890c-4f28-a4da-98424ffde2e3-kube-api-access-69jsb\") pod \"proxy-routing-7bea488f-sink-55999bf9fd-wx7hk\" (UID: \"46fcdc88-890c-4f28-a4da-98424ffde2e3\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:43:08.246081 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.246046    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jsb\" (UniqueName: \"kubernetes.io/projected/46fcdc88-890c-4f28-a4da-98424ffde2e3-kube-api-access-69jsb\") pod \"proxy-routing-7bea488f-sink-55999bf9fd-wx7hk\" (UID: \"46fcdc88-890c-4f28-a4da-98424ffde2e3\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:43:08.253891 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.253863    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jsb\" (UniqueName: \"kubernetes.io/projected/46fcdc88-890c-4f28-a4da-98424ffde2e3-kube-api-access-69jsb\") pod \"proxy-routing-7bea488f-sink-55999bf9fd-wx7hk\" (UID: \"46fcdc88-890c-4f28-a4da-98424ffde2e3\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:43:08.373110 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.373019    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:43:08.641345 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.641325    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"]
Oct 31 07:43:08.643945 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:43:08.643917    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fcdc88_890c_4f28_a4da_98424ffde2e3.slice/crio-6782e16536e932efac7dc06f39ae40225e637449b9a6dd5b48e129b7535692d5 WatchSource:0}: Error finding container 6782e16536e932efac7dc06f39ae40225e637449b9a6dd5b48e129b7535692d5: Status 404 returned error can't find the container with id 6782e16536e932efac7dc06f39ae40225e637449b9a6dd5b48e129b7535692d5
Oct 31 07:43:08.646376 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.646360    4913 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:43:08.884086 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:08.884048    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk" event={"ID":"46fcdc88-890c-4f28-a4da-98424ffde2e3","Type":"ContainerStarted","Data":"6782e16536e932efac7dc06f39ae40225e637449b9a6dd5b48e129b7535692d5"}
Oct 31 07:43:10.896424 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:10.896391    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk" event={"ID":"46fcdc88-890c-4f28-a4da-98424ffde2e3","Type":"ContainerStarted","Data":"049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee"}
Oct 31 07:43:10.896805 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:10.896597    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:43:10.897821 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:10.897783    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk" podUID="46fcdc88-890c-4f28-a4da-98424ffde2e3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/\": dial tcp 10.132.0.37:8080: connect: connection refused"
Oct 31 07:43:10.913669 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:10.913618    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk" podStartSLOduration=0.891810677 podStartE2EDuration="2.913604677s" totalImagesPullingTime="2.021794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:10.91175662 +0000 UTC m=+839.601524912" watchObservedRunningTime="2026-10-31 07:43:10.913604677 +0000 UTC m=+839.603372968"
Oct 31 07:43:11.901452 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:11.901378    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:43:15.469577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.469545    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"]
Oct 31 07:43:15.474678 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.474648    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.477608 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.477584    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-7bea488f-kube-rbac-proxy-sar-config\""
Oct 31 07:43:15.481105 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.481077    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-7bea488f-dockercfg-dkg78\""
Oct 31 07:43:15.481883 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.481859    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-7bea488f-predictor-serving-cert\""
Oct 31 07:43:15.483291 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.483265    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"]
Oct 31 07:43:15.605196 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.605159    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-rbac-proxy-sar-config\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.605413 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.605222    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxm5q\" (UniqueName: \"kubernetes.io/projected/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-api-access-mxm5q\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.605413 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.605341    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1826efaa-ce6a-433e-a18b-00c41f50f902-proxy-tls\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.706559 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.706519    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1826efaa-ce6a-433e-a18b-00c41f50f902-proxy-tls\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.706729 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.706576    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-rbac-proxy-sar-config\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.706729 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.706608    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxm5q\" (UniqueName: \"kubernetes.io/projected/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-api-access-mxm5q\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.707330 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.707305    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-rbac-proxy-sar-config\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.708968 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.708940    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1826efaa-ce6a-433e-a18b-00c41f50f902-proxy-tls\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:15.716492 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:15.716470    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxm5q\" (UniqueName: \"kubernetes.io/projected/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-api-access-mxm5q\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.133568 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.133526    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.138930 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.138900    4913 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") device path: \"\"" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.234356 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.234310    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.234540 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.234450    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") DevicePath \"\"" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.239034 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.239008    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") DevicePath \"csi-13aeae56f3f0b0c7f4061844cd414a617335ecd3fc49f28a4ecc747f75243906\"" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.356340 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.356302    4913 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7db6983c545fbe44d4669145a7380c9e2849df8625be2aaf27055dee4c22a107/globalmount\"" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.370284 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.370258    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.389670 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.389615    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.402258 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:43:19.402230    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:43:19.670387 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.670358    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"]
Oct 31 07:43:19.673063 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:43:19.673032    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1826efaa_ce6a_433e_a18b_00c41f50f902.slice/crio-b0667e8a81f56e19399aa15b82c7d0bd05237ee6b1f36b3bf038763ba8cf98be WatchSource:0}: Error finding container b0667e8a81f56e19399aa15b82c7d0bd05237ee6b1f36b3bf038763ba8cf98be: Status 404 returned error can't find the container with id b0667e8a81f56e19399aa15b82c7d0bd05237ee6b1f36b3bf038763ba8cf98be
Oct 31 07:43:19.933803 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.933764    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" event={"ID":"1826efaa-ce6a-433e-a18b-00c41f50f902","Type":"ContainerStarted","Data":"a3a5d08291326b1c97632afc8dbfe472f3f4c6328f5425ad3b73d84491e51f77"}
Oct 31 07:43:19.933989 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.933824    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.933989 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.933849    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" event={"ID":"1826efaa-ce6a-433e-a18b-00c41f50f902","Type":"ContainerStarted","Data":"62413a0fb106674b726741380447929fa6b29eb9c265a2274f2e92ad57315d9a"}
Oct 31 07:43:19.933989 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.933871    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" event={"ID":"1826efaa-ce6a-433e-a18b-00c41f50f902","Type":"ContainerStarted","Data":"b0667e8a81f56e19399aa15b82c7d0bd05237ee6b1f36b3bf038763ba8cf98be"}
Oct 31 07:43:19.933989 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.933894    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:19.935577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.935550    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/\": dial tcp 10.132.0.38:8080: connect: connection refused"
Oct 31 07:43:19.953405 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:19.953360    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podStartSLOduration=7.953346075 podStartE2EDuration="7.953346075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:19.952438023 +0000 UTC m=+848.642206315" watchObservedRunningTime="2026-10-31 07:43:19.953346075 +0000 UTC m=+848.643114367"
Oct 31 07:43:20.939314 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:20.939283    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:26.946068 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:26.946039    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:43:32.538880 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.538844    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"]
Oct 31 07:43:32.542382 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.542361    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.552783 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.552758    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"]
Oct 31 07:43:32.645547 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.645514    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5b9s\" (UniqueName: \"kubernetes.io/projected/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-api-access-c5b9s\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.645722 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.645568    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ac45971-5fd3-4330-bd9f-8d9beb079943-proxy-tls\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.645722 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.645667    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.645722 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.645715    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-rbac-proxy-sar-config\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.645846 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.645747    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") DevicePath \"csi-13aeae56f3f0b0c7f4061844cd414a617335ecd3fc49f28a4ecc747f75243906\"" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.649162 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.649140    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") DevicePath \"csi-13aeae56f3f0b0c7f4061844cd414a617335ecd3fc49f28a4ecc747f75243906\"" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.660844 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.660821    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.746261 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.746226    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-rbac-proxy-sar-config\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.746261 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.746264    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5b9s\" (UniqueName: \"kubernetes.io/projected/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-api-access-c5b9s\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.746533 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.746302    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ac45971-5fd3-4330-bd9f-8d9beb079943-proxy-tls\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.746967 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.746939    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-rbac-proxy-sar-config\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.748708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.748689    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ac45971-5fd3-4330-bd9f-8d9beb079943-proxy-tls\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.754138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.754116    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5b9s\" (UniqueName: \"kubernetes.io/projected/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-api-access-c5b9s\") pod \"proxy-routing-7bea488f-predictor-76b695dcf-jjzkv\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") " pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:32.859347 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:32.859269    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:33.129921 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:33.129838    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"]
Oct 31 07:43:33.133667 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:43:33.133638    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac45971_5fd3_4330_bd9f_8d9beb079943.slice/crio-8b26029b1a598b448f1ed05ba6c56c788c7096612e8b568eb23070714ad0170e WatchSource:0}: Error finding container 8b26029b1a598b448f1ed05ba6c56c788c7096612e8b568eb23070714ad0170e: Status 404 returned error can't find the container with id 8b26029b1a598b448f1ed05ba6c56c788c7096612e8b568eb23070714ad0170e
Oct 31 07:43:33.993820 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:33.993783    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" event={"ID":"8ac45971-5fd3-4330-bd9f-8d9beb079943","Type":"ContainerStarted","Data":"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37"}
Oct 31 07:43:33.994225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:33.993845    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" event={"ID":"8ac45971-5fd3-4330-bd9f-8d9beb079943","Type":"ContainerStarted","Data":"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26"}
Oct 31 07:43:33.994225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:33.993893    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" event={"ID":"8ac45971-5fd3-4330-bd9f-8d9beb079943","Type":"ContainerStarted","Data":"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b"}
Oct 31 07:43:33.994225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:33.993915    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" event={"ID":"8ac45971-5fd3-4330-bd9f-8d9beb079943","Type":"ContainerStarted","Data":"8b26029b1a598b448f1ed05ba6c56c788c7096612e8b568eb23070714ad0170e"}
Oct 31 07:43:33.994225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:33.993941    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:34.014078 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:34.014024    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podStartSLOduration=2.014009855 podStartE2EDuration="2.014009855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:34.012554982 +0000 UTC m=+862.702323272" watchObservedRunningTime="2026-10-31 07:43:34.014009855 +0000 UTC m=+862.703778147"
Oct 31 07:43:34.998557 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:34.998516    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:34.998919 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:34.998600    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:35.000945 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:35.000922    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:35.001341 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:35.001322    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:42.010366 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:42.010337    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:43:42.052801 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:42.052763    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"]
Oct 31 07:43:42.053255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:42.053035    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kserve-container" containerID="cri-o://62413a0fb106674b726741380447929fa6b29eb9c265a2274f2e92ad57315d9a" gracePeriod=30
Oct 31 07:43:42.053255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:42.053073    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy" containerID="cri-o://a3a5d08291326b1c97632afc8dbfe472f3f4c6328f5425ad3b73d84491e51f77" gracePeriod=30
Oct 31 07:43:43.033158 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:43.033124    4913 generic.go:356] "Generic (PLEG): container finished" podID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerID="a3a5d08291326b1c97632afc8dbfe472f3f4c6328f5425ad3b73d84491e51f77" exitCode=0
Oct 31 07:43:43.033561 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:43.033216    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" event={"ID":"1826efaa-ce6a-433e-a18b-00c41f50f902","Type":"ContainerDied","Data":"a3a5d08291326b1c97632afc8dbfe472f3f4c6328f5425ad3b73d84491e51f77"}
Oct 31 07:43:46.942936 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:46.942890    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 31 07:43:49.399904 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:43:49.399860    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:43:51.942833 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:51.942788    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 31 07:43:56.942642 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:56.942600    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 31 07:43:56.943038 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:43:56.942771    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:44:01.943325 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:01.943285    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 31 07:44:06.942932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:06.942885    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 31 07:44:11.490215 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:11.490171    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:44:11.491228 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:11.491202    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:44:11.943322 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:11.943280    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Oct 31 07:44:12.158852 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.158815    4913 generic.go:356] "Generic (PLEG): container finished" podID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerID="62413a0fb106674b726741380447929fa6b29eb9c265a2274f2e92ad57315d9a" exitCode=137
Oct 31 07:44:12.159007 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.158863    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" event={"ID":"1826efaa-ce6a-433e-a18b-00c41f50f902","Type":"ContainerDied","Data":"62413a0fb106674b726741380447929fa6b29eb9c265a2274f2e92ad57315d9a"}
Oct 31 07:44:12.250874 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.250843    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:44:12.381433 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.381399    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-rbac-proxy-sar-config\") pod \"1826efaa-ce6a-433e-a18b-00c41f50f902\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") "
Oct 31 07:44:12.381601 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.381555    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"1826efaa-ce6a-433e-a18b-00c41f50f902\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") "
Oct 31 07:44:12.381646 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.381616    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-api-access-mxm5q\" (UniqueName: \"kubernetes.io/projected/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-api-access-mxm5q\") pod \"1826efaa-ce6a-433e-a18b-00c41f50f902\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") "
Oct 31 07:44:12.381680 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.381648    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1826efaa-ce6a-433e-a18b-00c41f50f902-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1826efaa-ce6a-433e-a18b-00c41f50f902-proxy-tls\") pod \"1826efaa-ce6a-433e-a18b-00c41f50f902\" (UID: \"1826efaa-ce6a-433e-a18b-00c41f50f902\") "
Oct 31 07:44:12.381832 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.381805    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-rbac-proxy-sar-config" pod "1826efaa-ce6a-433e-a18b-00c41f50f902" (UID: "1826efaa-ce6a-433e-a18b-00c41f50f902"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:12.383744 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.383716    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-api-access-mxm5q" pod "1826efaa-ce6a-433e-a18b-00c41f50f902" (UID: "1826efaa-ce6a-433e-a18b-00c41f50f902"). InnerVolumeSpecName "kube-api-access-mxm5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:12.383744 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.383725    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1826efaa-ce6a-433e-a18b-00c41f50f902-proxy-tls" pod "1826efaa-ce6a-433e-a18b-00c41f50f902" (UID: "1826efaa-ce6a-433e-a18b-00c41f50f902"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:12.385754 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.385728    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5" pod "1826efaa-ce6a-433e-a18b-00c41f50f902" (UID: "1826efaa-ce6a-433e-a18b-00c41f50f902"). InnerVolumeSpecName "pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:44:12.483151 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.483068    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxm5q\" (UniqueName: \"kubernetes.io/projected/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-api-access-mxm5q\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:12.483151 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.483100    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1826efaa-ce6a-433e-a18b-00c41f50f902-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:12.483151 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:12.483112    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1826efaa-ce6a-433e-a18b-00c41f50f902-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:13.164908 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:13.164876    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw" event={"ID":"1826efaa-ce6a-433e-a18b-00c41f50f902","Type":"ContainerDied","Data":"b0667e8a81f56e19399aa15b82c7d0bd05237ee6b1f36b3bf038763ba8cf98be"}
Oct 31 07:44:13.165331 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:13.164915    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"
Oct 31 07:44:13.165331 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:13.164977    4913 scope.go:122] "RemoveContainer" containerID="a3a5d08291326b1c97632afc8dbfe472f3f4c6328f5425ad3b73d84491e51f77"
Oct 31 07:44:13.173972 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:13.173954    4913 scope.go:122] "RemoveContainer" containerID="62413a0fb106674b726741380447929fa6b29eb9c265a2274f2e92ad57315d9a"
Oct 31 07:44:13.190389 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:13.190364    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"]
Oct 31 07:44:13.195006 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:13.194983    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-6c85544c8f-8cgpw"]
Oct 31 07:44:13.419410 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:13.419342    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" path="/var/lib/kubelet/pods/1826efaa-ce6a-433e-a18b-00c41f50f902/volumes"
Oct 31 07:44:14.349479 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:14.349451    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-13001498cc_proxy-routing-7bea488f-sink-55999bf9fd-wx7hk_46fcdc88-890c-4f28-a4da-98424ffde2e3/kserve-container/0.log"
Oct 31 07:44:14.428692 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:14.428659    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"]
Oct 31 07:44:14.428867 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:14.428838    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk" podUID="46fcdc88-890c-4f28-a4da-98424ffde2e3" containerName="kserve-container" containerID="cri-o://049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee" gracePeriod=30
Oct 31 07:44:14.510689 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:14.510655    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"]
Oct 31 07:44:14.510946 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:14.510923    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kserve-container" containerID="cri-o://bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b" gracePeriod=30
Oct 31 07:44:14.511276 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:14.511234    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="agent" containerID="cri-o://adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37" gracePeriod=30
Oct 31 07:44:14.511597 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:14.511330    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy" containerID="cri-o://77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26" gracePeriod=30
Oct 31 07:44:14.999535 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:14.999498    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:44:15.176972 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:15.176935    4913 generic.go:356] "Generic (PLEG): container finished" podID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerID="77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26" exitCode=0
Oct 31 07:44:15.177136 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:15.177006    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" event={"ID":"8ac45971-5fd3-4330-bd9f-8d9beb079943","Type":"ContainerDied","Data":"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26"}
Oct 31 07:44:17.006908 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:17.006863    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 31 07:44:19.399735 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:44:19.399692    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:44:22.006910 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:22.006868    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 31 07:44:24.999129 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:24.999085    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:44:27.007198 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:27.007137    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 31 07:44:27.007597 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:27.007296    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:44:32.006869 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:32.006821    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 31 07:44:34.999785 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:34.999742    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:44:35.000277 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:34.999916    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:44:37.006406 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:37.006361    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 31 07:44:42.006264 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:42.006218    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Oct 31 07:44:44.621743 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.621712    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:44:44.684982 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.684926    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:44:44.750904 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.750870    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fcdc88-890c-4f28-a4da-98424ffde2e3-kube-api-access-69jsb\" (UniqueName: \"kubernetes.io/projected/46fcdc88-890c-4f28-a4da-98424ffde2e3-kube-api-access-69jsb\") pod \"46fcdc88-890c-4f28-a4da-98424ffde2e3\" (UID: \"46fcdc88-890c-4f28-a4da-98424ffde2e3\") "
Oct 31 07:44:44.752954 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.752926    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fcdc88-890c-4f28-a4da-98424ffde2e3-kube-api-access-69jsb" pod "46fcdc88-890c-4f28-a4da-98424ffde2e3" (UID: "46fcdc88-890c-4f28-a4da-98424ffde2e3"). InnerVolumeSpecName "kube-api-access-69jsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:44.852167 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.852131    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") pod \"8ac45971-5fd3-4330-bd9f-8d9beb079943\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") "
Oct 31 07:44:44.852167 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.852172    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac45971-5fd3-4330-bd9f-8d9beb079943-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ac45971-5fd3-4330-bd9f-8d9beb079943-proxy-tls\") pod \"8ac45971-5fd3-4330-bd9f-8d9beb079943\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") "
Oct 31 07:44:44.852378 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.852236    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-api-access-c5b9s\" (UniqueName: \"kubernetes.io/projected/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-api-access-c5b9s\") pod \"8ac45971-5fd3-4330-bd9f-8d9beb079943\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") "
Oct 31 07:44:44.852378 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.852289    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-rbac-proxy-sar-config\") pod \"8ac45971-5fd3-4330-bd9f-8d9beb079943\" (UID: \"8ac45971-5fd3-4330-bd9f-8d9beb079943\") "
Oct 31 07:44:44.852462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.852449    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69jsb\" (UniqueName: \"kubernetes.io/projected/46fcdc88-890c-4f28-a4da-98424ffde2e3-kube-api-access-69jsb\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:44.852780 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.852719    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-rbac-proxy-sar-config" pod "8ac45971-5fd3-4330-bd9f-8d9beb079943" (UID: "8ac45971-5fd3-4330-bd9f-8d9beb079943"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:44.854495 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.854466    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac45971-5fd3-4330-bd9f-8d9beb079943-proxy-tls" pod "8ac45971-5fd3-4330-bd9f-8d9beb079943" (UID: "8ac45971-5fd3-4330-bd9f-8d9beb079943"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:44.854602 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.854506    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-api-access-c5b9s" pod "8ac45971-5fd3-4330-bd9f-8d9beb079943" (UID: "8ac45971-5fd3-4330-bd9f-8d9beb079943"). InnerVolumeSpecName "kube-api-access-c5b9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:44.854852 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.854834    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5" pod "8ac45971-5fd3-4330-bd9f-8d9beb079943" (UID: "8ac45971-5fd3-4330-bd9f-8d9beb079943"). InnerVolumeSpecName "pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:44:44.952905 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.952816    4913 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" "
Oct 31 07:44:44.952905 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.952853    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ac45971-5fd3-4330-bd9f-8d9beb079943-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:44.952905 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.952867    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5b9s\" (UniqueName: \"kubernetes.io/projected/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-api-access-c5b9s\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:44.952905 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.952879    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ac45971-5fd3-4330-bd9f-8d9beb079943-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:44:44.964015 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:44.963995    4913 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5") on node "ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:44:45.053345 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.053307    4913 reconciler_common.go:299] "Volume detached for volume \"pvc-e6cf63d2-4e5a-427b-b068-876a4eca0bcd\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e0da2f279818c6a5\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"csi-13aeae56f3f0b0c7f4061844cd414a617335ecd3fc49f28a4ecc747f75243906\""
Oct 31 07:44:45.307641 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.307557    4913 generic.go:356] "Generic (PLEG): container finished" podID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerID="adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37" exitCode=137
Oct 31 07:44:45.307641 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.307588    4913 generic.go:356] "Generic (PLEG): container finished" podID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerID="bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b" exitCode=137
Oct 31 07:44:45.307641 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.307610    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"
Oct 31 07:44:45.307861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.307647    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" event={"ID":"8ac45971-5fd3-4330-bd9f-8d9beb079943","Type":"ContainerDied","Data":"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37"}
Oct 31 07:44:45.307861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.307749    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" event={"ID":"8ac45971-5fd3-4330-bd9f-8d9beb079943","Type":"ContainerDied","Data":"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b"}
Oct 31 07:44:45.307861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.307787    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv" event={"ID":"8ac45971-5fd3-4330-bd9f-8d9beb079943","Type":"ContainerDied","Data":"8b26029b1a598b448f1ed05ba6c56c788c7096612e8b568eb23070714ad0170e"}
Oct 31 07:44:45.307861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.307816    4913 scope.go:122] "RemoveContainer" containerID="adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37"
Oct 31 07:44:45.308974 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.308947    4913 generic.go:356] "Generic (PLEG): container finished" podID="46fcdc88-890c-4f28-a4da-98424ffde2e3" containerID="049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee" exitCode=137
Oct 31 07:44:45.309106 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.308985    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"
Oct 31 07:44:45.309106 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.308983    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk" event={"ID":"46fcdc88-890c-4f28-a4da-98424ffde2e3","Type":"ContainerDied","Data":"049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee"}
Oct 31 07:44:45.309106 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.309035    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk" event={"ID":"46fcdc88-890c-4f28-a4da-98424ffde2e3","Type":"ContainerDied","Data":"6782e16536e932efac7dc06f39ae40225e637449b9a6dd5b48e129b7535692d5"}
Oct 31 07:44:45.316416 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.316400    4913 scope.go:122] "RemoveContainer" containerID="77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26"
Oct 31 07:44:45.324286 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.324268    4913 scope.go:122] "RemoveContainer" containerID="bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b"
Oct 31 07:44:45.332298 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.332276    4913 scope.go:122] "RemoveContainer" containerID="adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37"
Oct 31 07:44:45.332404 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.332307    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"]
Oct 31 07:44:45.332601 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:44:45.332574    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37\": container with ID starting with adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37 not found: ID does not exist" containerID="adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37"
Oct 31 07:44:45.332753 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.332612    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37"} err="failed to get container status \"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37\": rpc error: code = NotFound desc = could not find container \"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37\": container with ID starting with adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37 not found: ID does not exist"
Oct 31 07:44:45.332753 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.332635    4913 scope.go:122] "RemoveContainer" containerID="77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26"
Oct 31 07:44:45.333278 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:44:45.333253    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26\": container with ID starting with 77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26 not found: ID does not exist" containerID="77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26"
Oct 31 07:44:45.333370 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.333286    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26"} err="failed to get container status \"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26\": rpc error: code = NotFound desc = could not find container \"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26\": container with ID starting with 77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26 not found: ID does not exist"
Oct 31 07:44:45.333370 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.333305    4913 scope.go:122] "RemoveContainer" containerID="bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b"
Oct 31 07:44:45.333571 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:44:45.333549    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b\": container with ID starting with bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b not found: ID does not exist" containerID="bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b"
Oct 31 07:44:45.333640 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.333579    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b"} err="failed to get container status \"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b\": rpc error: code = NotFound desc = could not find container \"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b\": container with ID starting with bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b not found: ID does not exist"
Oct 31 07:44:45.333640 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.333600    4913 scope.go:122] "RemoveContainer" containerID="adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37"
Oct 31 07:44:45.333816 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.333796    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-sink-55999bf9fd-wx7hk"]
Oct 31 07:44:45.333912 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.333841    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37"} err="failed to get container status \"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37\": rpc error: code = NotFound desc = could not find container \"adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37\": container with ID starting with adf5ab9ba20184a41dc7b57c692a8ae53d43ceaedba4b79ec7480af4de07fd37 not found: ID does not exist"
Oct 31 07:44:45.333912 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.333857    4913 scope.go:122] "RemoveContainer" containerID="77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26"
Oct 31 07:44:45.334116 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.334097    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26"} err="failed to get container status \"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26\": rpc error: code = NotFound desc = could not find container \"77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26\": container with ID starting with 77eef8a570cdd17d4e903a6f39670983cef15a726dccbde4db446b4e2277ed26 not found: ID does not exist"
Oct 31 07:44:45.334211 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.334118    4913 scope.go:122] "RemoveContainer" containerID="bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b"
Oct 31 07:44:45.334359 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.334342    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b"} err="failed to get container status \"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b\": rpc error: code = NotFound desc = could not find container \"bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b\": container with ID starting with bd17c64c109facde25af5619dc6ce4fd683335a022b88a1837226388296d5c0b not found: ID does not exist"
Oct 31 07:44:45.334435 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.334363    4913 scope.go:122] "RemoveContainer" containerID="049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee"
Oct 31 07:44:45.341579 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.341559    4913 scope.go:122] "RemoveContainer" containerID="049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee"
Oct 31 07:44:45.341847 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:44:45.341820    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee\": container with ID starting with 049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee not found: ID does not exist" containerID="049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee"
Oct 31 07:44:45.341910 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.341846    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee"} err="failed to get container status \"049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee\": rpc error: code = NotFound desc = could not find container \"049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee\": container with ID starting with 049799e68ad71bfc2492bb7bba83f334fcf7a991435224ed08282c21b0dc19ee not found: ID does not exist"
Oct 31 07:44:45.344797 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.344776    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"]
Oct 31 07:44:45.348775 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.348754    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-7bea488f-predictor-76b695dcf-jjzkv"]
Oct 31 07:44:45.419105 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.419066    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fcdc88-890c-4f28-a4da-98424ffde2e3" path="/var/lib/kubelet/pods/46fcdc88-890c-4f28-a4da-98424ffde2e3/volumes"
Oct 31 07:44:45.419594 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:45.419577    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" path="/var/lib/kubelet/pods/8ac45971-5fd3-4330-bd9f-8d9beb079943/volumes"
Oct 31 07:44:47.427799 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.427767    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"]
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428084    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kserve-container"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428094    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kserve-container"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428107    4913 cpu_manager.go:414] "containerMap: removing container" podUID="46fcdc88-890c-4f28-a4da-98424ffde2e3" containerName="kserve-container"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428113    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fcdc88-890c-4f28-a4da-98424ffde2e3" containerName="kserve-container"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428120    4913 cpu_manager.go:414] "containerMap: removing container" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kserve-container"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428124    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kserve-container"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428134    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428139    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428147    4913 cpu_manager.go:414] "containerMap: removing container" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428152    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428162    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="agent"
Oct 31 07:44:47.428172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428167    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="agent"
Oct 31 07:44:47.428538 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428257    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kserve-container"
Oct 31 07:44:47.428538 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428268    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fcdc88-890c-4f28-a4da-98424ffde2e3" containerName="kserve-container"
Oct 31 07:44:47.428538 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428275    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kserve-container"
Oct 31 07:44:47.428538 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428282    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="kube-rbac-proxy"
Oct 31 07:44:47.428538 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428288    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="1826efaa-ce6a-433e-a18b-00c41f50f902" containerName="kube-rbac-proxy"
Oct 31 07:44:47.428538 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.428293    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac45971-5fd3-4330-bd9f-8d9beb079943" containerName="agent"
Oct 31 07:44:47.432819 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.432793    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:44:47.440287 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.440132    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:44:47.440287 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.440204    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:44:47.440941 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.440914    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:44:47.442604 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.442583    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"]
Oct 31 07:44:47.574537 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.574500    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpsmz\" (UniqueName: \"kubernetes.io/projected/86d652a7-3c15-44bc-9cf5-e30f6ea96438-kube-api-access-hpsmz\") pod \"proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x\" (UID: \"86d652a7-3c15-44bc-9cf5-e30f6ea96438\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:44:47.675842 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.675799    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpsmz\" (UniqueName: \"kubernetes.io/projected/86d652a7-3c15-44bc-9cf5-e30f6ea96438-kube-api-access-hpsmz\") pod \"proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x\" (UID: \"86d652a7-3c15-44bc-9cf5-e30f6ea96438\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:44:47.684051 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.683987    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpsmz\" (UniqueName: \"kubernetes.io/projected/86d652a7-3c15-44bc-9cf5-e30f6ea96438-kube-api-access-hpsmz\") pod \"proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x\" (UID: \"86d652a7-3c15-44bc-9cf5-e30f6ea96438\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:44:47.754322 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:47.754286    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:44:48.022416 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:48.022392    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"]
Oct 31 07:44:48.024972 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:44:48.024944    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d652a7_3c15_44bc_9cf5_e30f6ea96438.slice/crio-488db9f37ab91fbdba3f0bb30986aa9c793e540b1247072c847c3ce45f2fc88a WatchSource:0}: Error finding container 488db9f37ab91fbdba3f0bb30986aa9c793e540b1247072c847c3ce45f2fc88a: Status 404 returned error can't find the container with id 488db9f37ab91fbdba3f0bb30986aa9c793e540b1247072c847c3ce45f2fc88a
Oct 31 07:44:48.326818 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:48.326738    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x" event={"ID":"86d652a7-3c15-44bc-9cf5-e30f6ea96438","Type":"ContainerStarted","Data":"9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1"}
Oct 31 07:44:48.326818 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:48.326791    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x" event={"ID":"86d652a7-3c15-44bc-9cf5-e30f6ea96438","Type":"ContainerStarted","Data":"488db9f37ab91fbdba3f0bb30986aa9c793e540b1247072c847c3ce45f2fc88a"}
Oct 31 07:44:48.327059 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:48.327036    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:44:48.328348 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:48.328320    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x" podUID="86d652a7-3c15-44bc-9cf5-e30f6ea96438" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/\": dial tcp 10.132.0.40:8080: connect: connection refused"
Oct 31 07:44:48.340361 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:48.340306    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x" podStartSLOduration=1.340289493 podStartE2EDuration="1.340289493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:48.34002382 +0000 UTC m=+937.029792115" watchObservedRunningTime="2026-10-31 07:44:48.340289493 +0000 UTC m=+937.030057786"
Oct 31 07:44:49.332456 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:49.332425    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:44:49.400018 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:44:49.399979    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:44:51.746322 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.746292    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"]
Oct 31 07:44:51.749819 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.749798    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.752976 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.752953    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-3070c2fe-kube-rbac-proxy-sar-config\""
Oct 31 07:44:51.761648 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.761592    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"]
Oct 31 07:44:51.799367 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.799340    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-3070c2fe-dockercfg-4gq85\""
Oct 31 07:44:51.799367 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.799352    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-3070c2fe-predictor-serving-cert\""
Oct 31 07:44:51.806239 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.806215    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-proxy-tls\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.806391 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.806263    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.806391 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.806378    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89cg5\" (UniqueName: \"kubernetes.io/projected/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-api-access-89cg5\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.907257 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.907212    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-proxy-tls\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.907442 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.907271    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.907442 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.907304    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89cg5\" (UniqueName: \"kubernetes.io/projected/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-api-access-89cg5\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.908016 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.907995    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.909642 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.909615    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-proxy-tls\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:44:51.915077 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:44:51.915050    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89cg5\" (UniqueName: \"kubernetes.io/projected/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-api-access-89cg5\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.164178 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.164127    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.169547 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.169516    4913 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") device path: \"\"" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.265511 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.265473    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.265688 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.265612    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") DevicePath \"\"" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.268337 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.268316    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") DevicePath \"csi-f5f26e8efe485d5af2def60cc6175a3de461ae8a8e06a0fcbfcacfec5da4b05b\"" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.434618 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.434590    4913 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d69d4c733dd3d054a3d1bf2dad3bc6f760cf9b7992d6f46a8621e4c9877bab37/globalmount\"" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.447277 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.447248    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-5b664c8785-b27ln\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.467224 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.467174    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:00.736343 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:00.736137    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"]
Oct 31 07:45:00.739109 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:45:00.739082    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1f91cb_c4f6_48b3_9c7b_0a74133890e9.slice/crio-aab05252a7dd5dd9c9d709cb06b4d80c9c3405d74a9a9b2f2255c9fe013a62af WatchSource:0}: Error finding container aab05252a7dd5dd9c9d709cb06b4d80c9c3405d74a9a9b2f2255c9fe013a62af: Status 404 returned error can't find the container with id aab05252a7dd5dd9c9d709cb06b4d80c9c3405d74a9a9b2f2255c9fe013a62af
Oct 31 07:45:01.381197 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:01.381143    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" event={"ID":"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9","Type":"ContainerStarted","Data":"c2d78e2336e3265898f3b250fe638e6e55c65072e9b29b03405accd2fc4bf928"}
Oct 31 07:45:01.381609 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:01.381234    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" event={"ID":"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9","Type":"ContainerStarted","Data":"c0a2d184011abec275c64ae5e8db8d0d32b0811e1a84dd496505cc96f27195f2"}
Oct 31 07:45:01.381609 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:01.381271    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:01.381609 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:01.381297    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" event={"ID":"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9","Type":"ContainerStarted","Data":"aab05252a7dd5dd9c9d709cb06b4d80c9c3405d74a9a9b2f2255c9fe013a62af"}
Oct 31 07:45:01.398743 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:01.398694    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podStartSLOduration=12.398680043 podStartE2EDuration="12.398680043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:01.398104439 +0000 UTC m=+950.087872730" watchObservedRunningTime="2026-10-31 07:45:01.398680043 +0000 UTC m=+950.088448335"
Oct 31 07:45:02.385444 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:02.385412    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:02.387422 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:02.387404    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:09.398207 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:09.398151    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:09.864261 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:09.864223    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"]
Oct 31 07:45:09.867921 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:09.867903    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:09.876537 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:09.876503    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"]
Oct 31 07:45:10.048015 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.047980    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.048235 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.048045    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.048235 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.048163    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4a0596-cc24-4ade-8462-e53222c5bee1-proxy-tls\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.048373 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.048255    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") DevicePath \"csi-f5f26e8efe485d5af2def60cc6175a3de461ae8a8e06a0fcbfcacfec5da4b05b\"" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.048373 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.048267    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvvj\" (UniqueName: \"kubernetes.io/projected/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-api-access-cbvvj\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.050639 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.050617    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") DevicePath \"csi-f5f26e8efe485d5af2def60cc6175a3de461ae8a8e06a0fcbfcacfec5da4b05b\"" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.061411 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.061388    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.149020 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.148930    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.149020 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.148999    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4a0596-cc24-4ade-8462-e53222c5bee1-proxy-tls\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.149246 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.149032    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvvj\" (UniqueName: \"kubernetes.io/projected/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-api-access-cbvvj\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.149703 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.149667    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-rbac-proxy-sar-config\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.151497 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.151478    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4a0596-cc24-4ade-8462-e53222c5bee1-proxy-tls\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.156436 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.156416    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvvj\" (UniqueName: \"kubernetes.io/projected/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-api-access-cbvvj\") pod \"proxy-routing-3070c2fe-predictor-76f458646f-b4g6m\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") " pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.185325 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.185289    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:10.460575 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:10.460548    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"]
Oct 31 07:45:10.463087 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:45:10.463055    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4a0596_cc24_4ade_8462_e53222c5bee1.slice/crio-18c4c58653e0133da92f90f989ebfbc6651cf6f9943e121994a221629aa4ad0b WatchSource:0}: Error finding container 18c4c58653e0133da92f90f989ebfbc6651cf6f9943e121994a221629aa4ad0b: Status 404 returned error can't find the container with id 18c4c58653e0133da92f90f989ebfbc6651cf6f9943e121994a221629aa4ad0b
Oct 31 07:45:11.424740 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:11.424709    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" event={"ID":"dd4a0596-cc24-4ade-8462-e53222c5bee1","Type":"ContainerStarted","Data":"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e"}
Oct 31 07:45:11.424899 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:11.424762    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" event={"ID":"dd4a0596-cc24-4ade-8462-e53222c5bee1","Type":"ContainerStarted","Data":"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12"}
Oct 31 07:45:11.424899 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:11.424788    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" event={"ID":"dd4a0596-cc24-4ade-8462-e53222c5bee1","Type":"ContainerStarted","Data":"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f"}
Oct 31 07:45:11.424899 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:11.424816    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:11.424899 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:11.424837    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" event={"ID":"dd4a0596-cc24-4ade-8462-e53222c5bee1","Type":"ContainerStarted","Data":"18c4c58653e0133da92f90f989ebfbc6651cf6f9943e121994a221629aa4ad0b"}
Oct 31 07:45:11.424899 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:11.424857    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:11.427309 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:11.427284    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:11.446600 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:11.446556    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podStartSLOduration=2.446542669 podStartE2EDuration="2.446542669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:11.445010171 +0000 UTC m=+960.134778467" watchObservedRunningTime="2026-10-31 07:45:11.446542669 +0000 UTC m=+960.136310961"
Oct 31 07:45:12.429646 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:12.429614    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:12.431295 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:12.431270    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:19.400705 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:45:19.400666    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:45:19.443594 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:19.443564    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:45:19.488114 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:19.488079    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"]
Oct 31 07:45:19.488362 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:19.488316    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kserve-container" containerID="cri-o://c0a2d184011abec275c64ae5e8db8d0d32b0811e1a84dd496505cc96f27195f2" gracePeriod=30
Oct 31 07:45:19.488502 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:19.488363    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy" containerID="cri-o://c2d78e2336e3265898f3b250fe638e6e55c65072e9b29b03405accd2fc4bf928" gracePeriod=30
Oct 31 07:45:20.464262 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:20.464222    4913 generic.go:356] "Generic (PLEG): container finished" podID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerID="c2d78e2336e3265898f3b250fe638e6e55c65072e9b29b03405accd2fc4bf928" exitCode=0
Oct 31 07:45:20.464644 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:20.464293    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" event={"ID":"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9","Type":"ContainerDied","Data":"c2d78e2336e3265898f3b250fe638e6e55c65072e9b29b03405accd2fc4bf928"}
Oct 31 07:45:24.395353 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:24.395306    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Oct 31 07:45:29.394951 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:29.394911    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Oct 31 07:45:34.395160 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:34.395112    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Oct 31 07:45:34.395569 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:34.395286    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:39.394935 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:39.394890    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Oct 31 07:45:44.394636 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:44.394592    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Oct 31 07:45:49.394712 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.394667    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Oct 31 07:45:49.400623 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:45:49.400591    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:45:49.587742 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.587709    4913 generic.go:356] "Generic (PLEG): container finished" podID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerID="c0a2d184011abec275c64ae5e8db8d0d32b0811e1a84dd496505cc96f27195f2" exitCode=137
Oct 31 07:45:49.587900 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.587780    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" event={"ID":"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9","Type":"ContainerDied","Data":"c0a2d184011abec275c64ae5e8db8d0d32b0811e1a84dd496505cc96f27195f2"}
Oct 31 07:45:49.672122 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.672097    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:49.801118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.801069    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-api-access-89cg5\" (UniqueName: \"kubernetes.io/projected/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-api-access-89cg5\") pod \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") "
Oct 31 07:45:49.801118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.801130    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-rbac-proxy-sar-config\") pod \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") "
Oct 31 07:45:49.801416 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.801308    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") "
Oct 31 07:45:49.801416 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.801361    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-proxy-tls\") pod \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\" (UID: \"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9\") "
Oct 31 07:45:49.801570 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.801547    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-rbac-proxy-sar-config" pod "8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" (UID: "8b1f91cb-c4f6-48b3-9c7b-0a74133890e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:49.801634 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.801616    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:45:49.803462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.803437    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-api-access-89cg5" pod "8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" (UID: "8b1f91cb-c4f6-48b3-9c7b-0a74133890e9"). InnerVolumeSpecName "kube-api-access-89cg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:49.803597 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.803573    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-proxy-tls" pod "8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" (UID: "8b1f91cb-c4f6-48b3-9c7b-0a74133890e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:49.804028 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.804009    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1" pod "8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" (UID: "8b1f91cb-c4f6-48b3-9c7b-0a74133890e9"). InnerVolumeSpecName "pvc-7542a182-33f1-4a2d-baed-a280d1569db7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:45:49.902906 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.902813    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89cg5\" (UniqueName: \"kubernetes.io/projected/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-kube-api-access-89cg5\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:45:49.902906 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:49.902851    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:45:50.597165 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:50.597135    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"
Oct 31 07:45:50.597675 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:50.597169    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln" event={"ID":"8b1f91cb-c4f6-48b3-9c7b-0a74133890e9","Type":"ContainerDied","Data":"aab05252a7dd5dd9c9d709cb06b4d80c9c3405d74a9a9b2f2255c9fe013a62af"}
Oct 31 07:45:50.597675 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:50.597248    4913 scope.go:122] "RemoveContainer" containerID="c2d78e2336e3265898f3b250fe638e6e55c65072e9b29b03405accd2fc4bf928"
Oct 31 07:45:50.606489 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:50.606472    4913 scope.go:122] "RemoveContainer" containerID="c0a2d184011abec275c64ae5e8db8d0d32b0811e1a84dd496505cc96f27195f2"
Oct 31 07:45:50.618947 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:50.618924    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"]
Oct 31 07:45:50.622702 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:50.622681    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-5b664c8785-b27ln"]
Oct 31 07:45:51.419661 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:51.419628    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" path="/var/lib/kubelet/pods/8b1f91cb-c4f6-48b3-9c7b-0a74133890e9/volumes"
Oct 31 07:45:56.695694 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:56.695661    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-13001498cc_proxy-routing-3070c2fe-predictor-76f458646f-b4g6m_dd4a0596-cc24-4ade-8462-e53222c5bee1/kserve-container/0.log"
Oct 31 07:45:56.775494 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:56.775461    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"]
Oct 31 07:45:56.775692 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:56.775657    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x" podUID="86d652a7-3c15-44bc-9cf5-e30f6ea96438" containerName="kserve-container" containerID="cri-o://9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1" gracePeriod=30
Oct 31 07:45:56.855756 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:56.855723    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"]
Oct 31 07:45:56.855985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:56.855958    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kserve-container" containerID="cri-o://b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f" gracePeriod=30
Oct 31 07:45:56.856076 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:56.855990    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="agent" containerID="cri-o://3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e" gracePeriod=30
Oct 31 07:45:56.856140 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:56.855983    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy" containerID="cri-o://10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12" gracePeriod=30
Oct 31 07:45:57.631172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:57.631132    4913 generic.go:356] "Generic (PLEG): container finished" podID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerID="10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12" exitCode=0
Oct 31 07:45:57.631360 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:57.631216    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" event={"ID":"dd4a0596-cc24-4ade-8462-e53222c5bee1","Type":"ContainerDied","Data":"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12"}
Oct 31 07:45:59.439955 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:45:59.439910    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 31 07:46:02.430434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:02.430391    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:46:04.439990 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:04.439950    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 31 07:46:09.439489 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:09.439446    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 31 07:46:09.439882 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:09.439568    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:46:12.431053 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:12.431002    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:46:14.439246 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:14.439176    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 31 07:46:19.400695 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:46:19.400657    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:46:19.439584 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:19.439546    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 31 07:46:22.430457 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:22.430407    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:46:22.430928 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:22.430524    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:46:24.439640 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:24.439595    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Oct 31 07:46:26.950304 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:26.950277    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:46:27.011851 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.011815    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d652a7-3c15-44bc-9cf5-e30f6ea96438-kube-api-access-hpsmz\" (UniqueName: \"kubernetes.io/projected/86d652a7-3c15-44bc-9cf5-e30f6ea96438-kube-api-access-hpsmz\") pod \"86d652a7-3c15-44bc-9cf5-e30f6ea96438\" (UID: \"86d652a7-3c15-44bc-9cf5-e30f6ea96438\") "
Oct 31 07:46:27.013736 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.013706    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d652a7-3c15-44bc-9cf5-e30f6ea96438-kube-api-access-hpsmz" pod "86d652a7-3c15-44bc-9cf5-e30f6ea96438" (UID: "86d652a7-3c15-44bc-9cf5-e30f6ea96438"). InnerVolumeSpecName "kube-api-access-hpsmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:27.046262 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.046240    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:46:27.112970 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.112937    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-api-access-cbvvj\" (UniqueName: \"kubernetes.io/projected/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-api-access-cbvvj\") pod \"dd4a0596-cc24-4ade-8462-e53222c5bee1\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") "
Oct 31 07:46:27.113155 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.112994    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd4a0596-cc24-4ade-8462-e53222c5bee1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4a0596-cc24-4ade-8462-e53222c5bee1-proxy-tls\") pod \"dd4a0596-cc24-4ade-8462-e53222c5bee1\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") "
Oct 31 07:46:27.113155 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.113069    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-rbac-proxy-sar-config\") pod \"dd4a0596-cc24-4ade-8462-e53222c5bee1\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") "
Oct 31 07:46:27.113306 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.113265    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpsmz\" (UniqueName: \"kubernetes.io/projected/86d652a7-3c15-44bc-9cf5-e30f6ea96438-kube-api-access-hpsmz\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:27.113472 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.113442    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-rbac-proxy-sar-config" pod "dd4a0596-cc24-4ade-8462-e53222c5bee1" (UID: "dd4a0596-cc24-4ade-8462-e53222c5bee1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:27.115053 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.115029    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4a0596-cc24-4ade-8462-e53222c5bee1-proxy-tls" pod "dd4a0596-cc24-4ade-8462-e53222c5bee1" (UID: "dd4a0596-cc24-4ade-8462-e53222c5bee1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:46:27.115121 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.115077    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-api-access-cbvvj" pod "dd4a0596-cc24-4ade-8462-e53222c5bee1" (UID: "dd4a0596-cc24-4ade-8462-e53222c5bee1"). InnerVolumeSpecName "kube-api-access-cbvvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:27.214209 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.214157    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") pod \"dd4a0596-cc24-4ade-8462-e53222c5bee1\" (UID: \"dd4a0596-cc24-4ade-8462-e53222c5bee1\") "
Oct 31 07:46:27.214375 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.214358    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:27.214418 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.214376    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvvj\" (UniqueName: \"kubernetes.io/projected/dd4a0596-cc24-4ade-8462-e53222c5bee1-kube-api-access-cbvvj\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:27.214418 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.214387    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd4a0596-cc24-4ade-8462-e53222c5bee1-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:46:27.216738 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.216716    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1" pod "dd4a0596-cc24-4ade-8462-e53222c5bee1" (UID: "dd4a0596-cc24-4ade-8462-e53222c5bee1"). InnerVolumeSpecName "pvc-7542a182-33f1-4a2d-baed-a280d1569db7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:46:27.315118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.315048    4913 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" "
Oct 31 07:46:27.325122 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.325102    4913 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-7542a182-33f1-4a2d-baed-a280d1569db7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1") on node "ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:46:27.415442 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.415401    4913 reconciler_common.go:299] "Volume detached for volume \"pvc-7542a182-33f1-4a2d-baed-a280d1569db7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b8f886de065315d1\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"csi-f5f26e8efe485d5af2def60cc6175a3de461ae8a8e06a0fcbfcacfec5da4b05b\""
Oct 31 07:46:27.755830 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.755801    4913 generic.go:356] "Generic (PLEG): container finished" podID="86d652a7-3c15-44bc-9cf5-e30f6ea96438" containerID="9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1" exitCode=137
Oct 31 07:46:27.756018 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.755850    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"
Oct 31 07:46:27.756018 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.755890    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x" event={"ID":"86d652a7-3c15-44bc-9cf5-e30f6ea96438","Type":"ContainerDied","Data":"9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1"}
Oct 31 07:46:27.756018 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.755945    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x" event={"ID":"86d652a7-3c15-44bc-9cf5-e30f6ea96438","Type":"ContainerDied","Data":"488db9f37ab91fbdba3f0bb30986aa9c793e540b1247072c847c3ce45f2fc88a"}
Oct 31 07:46:27.756018 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.755965    4913 scope.go:122] "RemoveContainer" containerID="9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1"
Oct 31 07:46:27.757927 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.757901    4913 generic.go:356] "Generic (PLEG): container finished" podID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerID="3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e" exitCode=0
Oct 31 07:46:27.757927 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.757927    4913 generic.go:356] "Generic (PLEG): container finished" podID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerID="b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f" exitCode=137
Oct 31 07:46:27.758116 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.757950    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" event={"ID":"dd4a0596-cc24-4ade-8462-e53222c5bee1","Type":"ContainerDied","Data":"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e"}
Oct 31 07:46:27.758116 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.757965    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"
Oct 31 07:46:27.758116 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.758009    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" event={"ID":"dd4a0596-cc24-4ade-8462-e53222c5bee1","Type":"ContainerDied","Data":"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f"}
Oct 31 07:46:27.758116 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.758048    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m" event={"ID":"dd4a0596-cc24-4ade-8462-e53222c5bee1","Type":"ContainerDied","Data":"18c4c58653e0133da92f90f989ebfbc6651cf6f9943e121994a221629aa4ad0b"}
Oct 31 07:46:27.764484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.764466    4913 scope.go:122] "RemoveContainer" containerID="9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1"
Oct 31 07:46:27.764762 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:46:27.764738    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1\": container with ID starting with 9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1 not found: ID does not exist" containerID="9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1"
Oct 31 07:46:27.764816 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.764766    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1"} err="failed to get container status \"9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1\": rpc error: code = NotFound desc = could not find container \"9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1\": container with ID starting with 9f44e9478cf1bd5e8acde2e9a7e0fb0e6df61e05d4f519f44d0be8ea7fb78df1 not found: ID does not exist"
Oct 31 07:46:27.764816 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.764783    4913 scope.go:122] "RemoveContainer" containerID="3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e"
Oct 31 07:46:27.771901 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.771884    4913 scope.go:122] "RemoveContainer" containerID="10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12"
Oct 31 07:46:27.774139 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.774115    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"]
Oct 31 07:46:27.777588 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.777569    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-sink-7ff9bc89bf-rmn4x"]
Oct 31 07:46:27.779872 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.779726    4913 scope.go:122] "RemoveContainer" containerID="b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f"
Oct 31 07:46:27.786930 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.786914    4913 scope.go:122] "RemoveContainer" containerID="3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e"
Oct 31 07:46:27.787204 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:46:27.787168    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e\": container with ID starting with 3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e not found: ID does not exist" containerID="3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e"
Oct 31 07:46:27.787294 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.787273    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e"} err="failed to get container status \"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e\": rpc error: code = NotFound desc = could not find container \"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e\": container with ID starting with 3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e not found: ID does not exist"
Oct 31 07:46:27.787365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.787295    4913 scope.go:122] "RemoveContainer" containerID="10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12"
Oct 31 07:46:27.787537 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:46:27.787519    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12\": container with ID starting with 10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12 not found: ID does not exist" containerID="10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12"
Oct 31 07:46:27.787597 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.787540    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12"} err="failed to get container status \"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12\": rpc error: code = NotFound desc = could not find container \"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12\": container with ID starting with 10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12 not found: ID does not exist"
Oct 31 07:46:27.787597 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.787552    4913 scope.go:122] "RemoveContainer" containerID="b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f"
Oct 31 07:46:27.787734 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.787712    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"]
Oct 31 07:46:27.787817 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:46:27.787764    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f\": container with ID starting with b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f not found: ID does not exist" containerID="b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f"
Oct 31 07:46:27.787817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.787783    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f"} err="failed to get container status \"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f\": rpc error: code = NotFound desc = could not find container \"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f\": container with ID starting with b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f not found: ID does not exist"
Oct 31 07:46:27.787817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.787796    4913 scope.go:122] "RemoveContainer" containerID="3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e"
Oct 31 07:46:27.788026 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.788007    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e"} err="failed to get container status \"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e\": rpc error: code = NotFound desc = could not find container \"3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e\": container with ID starting with 3c8b5c7decdab3843a445edfd9e5b4b53063da0625378f36ed16d01bbda0322e not found: ID does not exist"
Oct 31 07:46:27.788026 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.788022    4913 scope.go:122] "RemoveContainer" containerID="10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12"
Oct 31 07:46:27.788236 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.788216    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12"} err="failed to get container status \"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12\": rpc error: code = NotFound desc = could not find container \"10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12\": container with ID starting with 10ef32b9a844562803f6caae9fe8b34e432c953ea6660879892b589be880bd12 not found: ID does not exist"
Oct 31 07:46:27.788293 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.788238    4913 scope.go:122] "RemoveContainer" containerID="b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f"
Oct 31 07:46:27.788471 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.788449    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f"} err="failed to get container status \"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f\": rpc error: code = NotFound desc = could not find container \"b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f\": container with ID starting with b068feb075d65c52e307e702ba20e44f37f6a0a3ffb26b86413e144b9b1c840f not found: ID does not exist"
Oct 31 07:46:27.792499 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:27.792473    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-3070c2fe-predictor-76f458646f-b4g6m"]
Oct 31 07:46:29.418745 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.418713    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d652a7-3c15-44bc-9cf5-e30f6ea96438" path="/var/lib/kubelet/pods/86d652a7-3c15-44bc-9cf5-e30f6ea96438/volumes"
Oct 31 07:46:29.419129 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.419106    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" path="/var/lib/kubelet/pods/dd4a0596-cc24-4ade-8462-e53222c5bee1/volumes"
Oct 31 07:46:29.856314 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856277    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"]
Oct 31 07:46:29.856781 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856757    4913 cpu_manager.go:414] "containerMap: removing container" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kserve-container"
Oct 31 07:46:29.856861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856783    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kserve-container"
Oct 31 07:46:29.856861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856800    4913 cpu_manager.go:414] "containerMap: removing container" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy"
Oct 31 07:46:29.856861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856807    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy"
Oct 31 07:46:29.856861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856826    4913 cpu_manager.go:414] "containerMap: removing container" podUID="86d652a7-3c15-44bc-9cf5-e30f6ea96438" containerName="kserve-container"
Oct 31 07:46:29.856861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856835    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d652a7-3c15-44bc-9cf5-e30f6ea96438" containerName="kserve-container"
Oct 31 07:46:29.856861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856841    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kserve-container"
Oct 31 07:46:29.856861 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856846    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kserve-container"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856867    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856872    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856912    4913 cpu_manager.go:414] "containerMap: removing container" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="agent"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856919    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="agent"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856988    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="agent"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.856999    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kserve-container"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.857007    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4a0596-cc24-4ade-8462-e53222c5bee1" containerName="kube-rbac-proxy"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.857017    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d652a7-3c15-44bc-9cf5-e30f6ea96438" containerName="kserve-container"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.857025    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kserve-container"
Oct 31 07:46:29.857146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.857034    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1f91cb-c4f6-48b3-9c7b-0a74133890e9" containerName="kube-rbac-proxy"
Oct 31 07:46:29.860321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.860302    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:46:29.867162 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.867122    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"]
Oct 31 07:46:29.868868 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.868705    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:46:29.868868 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.868793    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:46:29.869667 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.869645    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:46:29.933619 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:29.933588    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvj9\" (UniqueName: \"kubernetes.io/projected/ef762d1b-c67f-44b7-ab32-5a3df91f13c5-kube-api-access-nvvj9\") pod \"proxy-routing-75d6970a-sink-7844dc4698-wd6m9\" (UID: \"ef762d1b-c67f-44b7-ab32-5a3df91f13c5\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:46:30.034698 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.034662    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvj9\" (UniqueName: \"kubernetes.io/projected/ef762d1b-c67f-44b7-ab32-5a3df91f13c5-kube-api-access-nvvj9\") pod \"proxy-routing-75d6970a-sink-7844dc4698-wd6m9\" (UID: \"ef762d1b-c67f-44b7-ab32-5a3df91f13c5\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:46:30.042369 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.042347    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvj9\" (UniqueName: \"kubernetes.io/projected/ef762d1b-c67f-44b7-ab32-5a3df91f13c5-kube-api-access-nvvj9\") pod \"proxy-routing-75d6970a-sink-7844dc4698-wd6m9\" (UID: \"ef762d1b-c67f-44b7-ab32-5a3df91f13c5\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:46:30.175168 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.175140    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:46:30.442287 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.442206    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"]
Oct 31 07:46:30.445082 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:46:30.445057    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef762d1b_c67f_44b7_ab32_5a3df91f13c5.slice/crio-9861ff3c54fa652922a4b8b98cb2fa143b51850f4af6dc1b53c29c1268f7fcfa WatchSource:0}: Error finding container 9861ff3c54fa652922a4b8b98cb2fa143b51850f4af6dc1b53c29c1268f7fcfa: Status 404 returned error can't find the container with id 9861ff3c54fa652922a4b8b98cb2fa143b51850f4af6dc1b53c29c1268f7fcfa
Oct 31 07:46:30.773832 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.773755    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9" event={"ID":"ef762d1b-c67f-44b7-ab32-5a3df91f13c5","Type":"ContainerStarted","Data":"bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb"}
Oct 31 07:46:30.773832 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.773802    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9" event={"ID":"ef762d1b-c67f-44b7-ab32-5a3df91f13c5","Type":"ContainerStarted","Data":"9861ff3c54fa652922a4b8b98cb2fa143b51850f4af6dc1b53c29c1268f7fcfa"}
Oct 31 07:46:30.774068 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.773954    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:46:30.776039 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.776023    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:46:30.787999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:30.787959    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9" podStartSLOduration=1.78794703 podStartE2EDuration="1.78794703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:46:30.787092774 +0000 UTC m=+1039.476861084" watchObservedRunningTime="2026-10-31 07:46:30.78794703 +0000 UTC m=+1039.477715322"
Oct 31 07:46:34.182894 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.182855    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"]
Oct 31 07:46:34.186439 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.186421    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.189554 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.189530    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-75d6970a-kube-rbac-proxy-sar-config\""
Oct 31 07:46:34.190201 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.190165    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-75d6970a-predictor-serving-cert\""
Oct 31 07:46:34.190301 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.190257    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-75d6970a-dockercfg-xtkr4\""
Oct 31 07:46:34.195544 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.195524    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"]
Oct 31 07:46:34.262987 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.262953    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-rbac-proxy-sar-config\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.263161 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.263003    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ed6857-c101-48a8-8bff-6bcb2706fe44-proxy-tls\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.263161 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.263072    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjg9t\" (UniqueName: \"kubernetes.io/projected/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-api-access-xjg9t\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.364379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.364344    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-rbac-proxy-sar-config\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.364574 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.364411    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ed6857-c101-48a8-8bff-6bcb2706fe44-proxy-tls\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.364574 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.364446    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjg9t\" (UniqueName: \"kubernetes.io/projected/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-api-access-xjg9t\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.365032 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.365011    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-rbac-proxy-sar-config\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.366913 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.366892    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ed6857-c101-48a8-8bff-6bcb2706fe44-proxy-tls\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:34.371725 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:34.371704    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjg9t\" (UniqueName: \"kubernetes.io/projected/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-api-access-xjg9t\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:35.635156 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.635123    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"]
Oct 31 07:46:35.639058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.639037    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.645475 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.645453    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Oct 31 07:46:35.647985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.647945    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:46:35.648118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.648033    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:46:35.648373 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.648353    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:46:35.648605 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.648365    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:46:35.648605 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.648413    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 31 07:46:35.648605 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.648464    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:46:35.650583 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.650564    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"]
Oct 31 07:46:35.776103 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.776069    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6250c4c7-c6fc-47af-bfaf-693020fb62c6-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.776278 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.776117    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.776278 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.776230    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.776405 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.776338    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vb8\" (UniqueName: \"kubernetes.io/projected/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-api-access-89vb8\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.776405 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.776385    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.877754 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.877720    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6250c4c7-c6fc-47af-bfaf-693020fb62c6-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.877754 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.877759    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.877967 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.877778    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.877967 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.877821    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vb8\" (UniqueName: \"kubernetes.io/projected/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-api-access-89vb8\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.877967 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.877845    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.878265 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.878237    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.878547 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.878526    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.878621 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.878551    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.880255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.880235    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6250c4c7-c6fc-47af-bfaf-693020fb62c6-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.888443 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.888383    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vb8\" (UniqueName: \"kubernetes.io/projected/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-api-access-89vb8\") pod \"isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") " pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:35.957321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:35.957280    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:46:36.228143 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:36.228093    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"]
Oct 31 07:46:36.230354 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:46:36.230322    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6250c4c7_c6fc_47af_bfaf_693020fb62c6.slice/crio-b8bd5be00c8e0a4ab3a0e49f67cf40542e6f6a64ba0fb6eca5e168d945bfb0b7 WatchSource:0}: Error finding container b8bd5be00c8e0a4ab3a0e49f67cf40542e6f6a64ba0fb6eca5e168d945bfb0b7: Status 404 returned error can't find the container with id b8bd5be00c8e0a4ab3a0e49f67cf40542e6f6a64ba0fb6eca5e168d945bfb0b7
Oct 31 07:46:36.798617 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:36.798576    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" event={"ID":"6250c4c7-c6fc-47af-bfaf-693020fb62c6","Type":"ContainerStarted","Data":"93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630"}
Oct 31 07:46:36.799125 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:36.798654    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" event={"ID":"6250c4c7-c6fc-47af-bfaf-693020fb62c6","Type":"ContainerStarted","Data":"b8bd5be00c8e0a4ab3a0e49f67cf40542e6f6a64ba0fb6eca5e168d945bfb0b7"}
Oct 31 07:46:40.818489 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:40.818403    4913 generic.go:356] "Generic (PLEG): container finished" podID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerID="93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630" exitCode=0
Oct 31 07:46:40.818489 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:40.818444    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" event={"ID":"6250c4c7-c6fc-47af-bfaf-693020fb62c6","Type":"ContainerDied","Data":"93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630"}
Oct 31 07:46:42.128483 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.128439    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:42.134923 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.134889    4913 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") device path: \"\"" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:42.229873 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.229481    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:42.229873 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.229655    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") DevicePath \"\"" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:42.233234 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.232715    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") DevicePath \"csi-cc9c81910fc929be31d8a648b35061fbfe6d1369d7d78e47204f437c3f0c0361\"" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:42.409578 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.409540    4913 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/f559187c6549875a5eb3da767c6c11a6a9361ffb560608c75622e8063aa3d352/globalmount\"" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:42.427599 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.427567    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") pod \"proxy-routing-75d6970a-predictor-8569ddb866-wzzt4\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") " pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:42.600741 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.600703    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:42.912963 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:42.912929    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"]
Oct 31 07:46:42.932312 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:46:42.932274    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ed6857_c101_48a8_8bff_6bcb2706fe44.slice/crio-1000961b080224b1bd597eb728b8419dd61f3a3fb9d39d23ffb1a2311b480484 WatchSource:0}: Error finding container 1000961b080224b1bd597eb728b8419dd61f3a3fb9d39d23ffb1a2311b480484: Status 404 returned error can't find the container with id 1000961b080224b1bd597eb728b8419dd61f3a3fb9d39d23ffb1a2311b480484
Oct 31 07:46:43.841060 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:43.841026    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" event={"ID":"b8ed6857-c101-48a8-8bff-6bcb2706fe44","Type":"ContainerStarted","Data":"50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148"}
Oct 31 07:46:43.841560 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:43.841097    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" event={"ID":"b8ed6857-c101-48a8-8bff-6bcb2706fe44","Type":"ContainerStarted","Data":"64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa"}
Oct 31 07:46:43.841560 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:43.841131    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" event={"ID":"b8ed6857-c101-48a8-8bff-6bcb2706fe44","Type":"ContainerStarted","Data":"1000961b080224b1bd597eb728b8419dd61f3a3fb9d39d23ffb1a2311b480484"}
Oct 31 07:46:43.842095 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:43.842075    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:43.842172 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:43.842120    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:43.844955 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:43.844895    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:46:43.862823 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:43.862767    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podStartSLOduration=12.862730964 podStartE2EDuration="12.862730964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:46:43.861196616 +0000 UTC m=+1052.550964927" watchObservedRunningTime="2026-10-31 07:46:43.862730964 +0000 UTC m=+1052.552499258"
Oct 31 07:46:49.407569 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:46:49.407530    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:46:50.861299 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:46:50.861265    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:47:02.948745 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:02.948704    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" event={"ID":"6250c4c7-c6fc-47af-bfaf-693020fb62c6","Type":"ContainerStarted","Data":"9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b"}
Oct 31 07:47:02.949155 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:02.948776    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" event={"ID":"6250c4c7-c6fc-47af-bfaf-693020fb62c6","Type":"ContainerStarted","Data":"3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba"}
Oct 31 07:47:02.949155 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:02.948964    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:47:02.949155 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:02.949015    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:47:02.967926 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:02.967874    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" podStartSLOduration=2.267924018 podStartE2EDuration="27.967859745s" totalImagesPullingTime="21.573922535s" totalInitContainerRuntime="4.126013192s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:47:02.967276446 +0000 UTC m=+1071.657044754" watchObservedRunningTime="2026-10-31 07:47:02.967859745 +0000 UTC m=+1071.657628037"
Oct 31 07:47:08.959542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:08.959508    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:47:19.399904 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:47:19.399864    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:47:22.365632 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:22.365598    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"]
Oct 31 07:47:22.366061 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:22.365811    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9" podUID="ef762d1b-c67f-44b7-ab32-5a3df91f13c5" containerName="kserve-container" containerID="cri-o://bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb" gracePeriod=30
Oct 31 07:47:22.443320 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:22.443274    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"]
Oct 31 07:47:22.443624 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:22.443574    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kserve-container" containerID="cri-o://64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa" gracePeriod=30
Oct 31 07:47:22.443720 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:22.443597    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy" containerID="cri-o://50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148" gracePeriod=30
Oct 31 07:47:23.032194 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:23.032158    4913 generic.go:356] "Generic (PLEG): container finished" podID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerID="50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148" exitCode=0
Oct 31 07:47:23.032382 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:23.032236    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" event={"ID":"b8ed6857-c101-48a8-8bff-6bcb2706fe44","Type":"ContainerDied","Data":"50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148"}
Oct 31 07:47:25.856295 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:25.856255    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 31 07:47:30.855714 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:30.855671    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 31 07:47:35.855777 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:35.855730    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 31 07:47:35.856278 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:35.855874    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:47:38.961061 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:38.961023    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 31 07:47:40.856154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:40.856112    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 31 07:47:45.856470 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:45.856419    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 31 07:47:48.960774 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:48.960731    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 31 07:47:49.400025 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:47:49.399938    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:47:50.855771 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:50.855732    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Oct 31 07:47:52.560122 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.560097    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:47:52.653039 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.653017    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:47:52.721160 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.721126    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef762d1b-c67f-44b7-ab32-5a3df91f13c5-kube-api-access-nvvj9\" (UniqueName: \"kubernetes.io/projected/ef762d1b-c67f-44b7-ab32-5a3df91f13c5-kube-api-access-nvvj9\") pod \"ef762d1b-c67f-44b7-ab32-5a3df91f13c5\" (UID: \"ef762d1b-c67f-44b7-ab32-5a3df91f13c5\") "
Oct 31 07:47:52.723150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.723130    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef762d1b-c67f-44b7-ab32-5a3df91f13c5-kube-api-access-nvvj9" pod "ef762d1b-c67f-44b7-ab32-5a3df91f13c5" (UID: "ef762d1b-c67f-44b7-ab32-5a3df91f13c5"). InnerVolumeSpecName "kube-api-access-nvvj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:47:52.822099 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.822070    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-rbac-proxy-sar-config\") pod \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") "
Oct 31 07:47:52.822273 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.822125    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ed6857-c101-48a8-8bff-6bcb2706fe44-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ed6857-c101-48a8-8bff-6bcb2706fe44-proxy-tls\") pod \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") "
Oct 31 07:47:52.822321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.822277    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") pod \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") "
Oct 31 07:47:52.822321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.822316    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-api-access-xjg9t\" (UniqueName: \"kubernetes.io/projected/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-api-access-xjg9t\") pod \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\" (UID: \"b8ed6857-c101-48a8-8bff-6bcb2706fe44\") "
Oct 31 07:47:52.822538 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.822509    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-rbac-proxy-sar-config" pod "b8ed6857-c101-48a8-8bff-6bcb2706fe44" (UID: "b8ed6857-c101-48a8-8bff-6bcb2706fe44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:47:52.822584 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.822533    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvj9\" (UniqueName: \"kubernetes.io/projected/ef762d1b-c67f-44b7-ab32-5a3df91f13c5-kube-api-access-nvvj9\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:47:52.824429 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.824396    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ed6857-c101-48a8-8bff-6bcb2706fe44-proxy-tls" pod "b8ed6857-c101-48a8-8bff-6bcb2706fe44" (UID: "b8ed6857-c101-48a8-8bff-6bcb2706fe44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:47:52.824547 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.824471    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-api-access-xjg9t" pod "b8ed6857-c101-48a8-8bff-6bcb2706fe44" (UID: "b8ed6857-c101-48a8-8bff-6bcb2706fe44"). InnerVolumeSpecName "kube-api-access-xjg9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:47:52.824942 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.824920    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f" pod "b8ed6857-c101-48a8-8bff-6bcb2706fe44" (UID: "b8ed6857-c101-48a8-8bff-6bcb2706fe44"). InnerVolumeSpecName "pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:47:52.923125 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.923093    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ed6857-c101-48a8-8bff-6bcb2706fe44-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:47:52.923305 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.923147    4913 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" "
Oct 31 07:47:52.923305 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.923162    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjg9t\" (UniqueName: \"kubernetes.io/projected/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-api-access-xjg9t\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:47:52.923305 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.923172    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ed6857-c101-48a8-8bff-6bcb2706fe44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:47:52.932920 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:52.932900    4913 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f") on node "ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:47:53.024365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.024294    4913 reconciler_common.go:299] "Volume detached for volume \"pvc-255196dd-5e98-4b37-9eff-10af9d7be4fb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0530ef8e0759a722f\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"csi-cc9c81910fc929be31d8a648b35061fbfe6d1369d7d78e47204f437c3f0c0361\""
Oct 31 07:47:53.159615 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.159579    4913 generic.go:356] "Generic (PLEG): container finished" podID="ef762d1b-c67f-44b7-ab32-5a3df91f13c5" containerID="bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb" exitCode=137
Oct 31 07:47:53.159805 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.159654    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"
Oct 31 07:47:53.159805 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.159666    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9" event={"ID":"ef762d1b-c67f-44b7-ab32-5a3df91f13c5","Type":"ContainerDied","Data":"bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb"}
Oct 31 07:47:53.159805 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.159753    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9" event={"ID":"ef762d1b-c67f-44b7-ab32-5a3df91f13c5","Type":"ContainerDied","Data":"9861ff3c54fa652922a4b8b98cb2fa143b51850f4af6dc1b53c29c1268f7fcfa"}
Oct 31 07:47:53.159805 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.159779    4913 scope.go:122] "RemoveContainer" containerID="bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb"
Oct 31 07:47:53.161571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.161546    4913 generic.go:356] "Generic (PLEG): container finished" podID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerID="64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa" exitCode=137
Oct 31 07:47:53.161706 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.161623    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"
Oct 31 07:47:53.161706 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.161637    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" event={"ID":"b8ed6857-c101-48a8-8bff-6bcb2706fe44","Type":"ContainerDied","Data":"64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa"}
Oct 31 07:47:53.161706 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.161683    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4" event={"ID":"b8ed6857-c101-48a8-8bff-6bcb2706fe44","Type":"ContainerDied","Data":"1000961b080224b1bd597eb728b8419dd61f3a3fb9d39d23ffb1a2311b480484"}
Oct 31 07:47:53.169909 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.169895    4913 scope.go:122] "RemoveContainer" containerID="bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb"
Oct 31 07:47:53.170322 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:47:53.170285    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb\": container with ID starting with bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb not found: ID does not exist" containerID="bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb"
Oct 31 07:47:53.170380 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.170327    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb"} err="failed to get container status \"bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb\": rpc error: code = NotFound desc = could not find container \"bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb\": container with ID starting with bd5ed9d9b54621456fc36753538ee5f666c42b8d734253afab174490a5c8a8bb not found: ID does not exist"
Oct 31 07:47:53.170380 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.170345    4913 scope.go:122] "RemoveContainer" containerID="50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148"
Oct 31 07:47:53.178027 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.178011    4913 scope.go:122] "RemoveContainer" containerID="64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa"
Oct 31 07:47:53.180484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.180462    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"]
Oct 31 07:47:53.183976 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.183955    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-sink-7844dc4698-wd6m9"]
Oct 31 07:47:53.186863 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.186839    4913 scope.go:122] "RemoveContainer" containerID="50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148"
Oct 31 07:47:53.187157 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:47:53.187137    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148\": container with ID starting with 50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148 not found: ID does not exist" containerID="50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148"
Oct 31 07:47:53.187251 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.187163    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148"} err="failed to get container status \"50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148\": rpc error: code = NotFound desc = could not find container \"50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148\": container with ID starting with 50c8e88b820c965f410c4080df66539dc97dd8476ac169614c4994219612f148 not found: ID does not exist"
Oct 31 07:47:53.187251 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.187202    4913 scope.go:122] "RemoveContainer" containerID="64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa"
Oct 31 07:47:53.187438 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:47:53.187423    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa\": container with ID starting with 64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa not found: ID does not exist" containerID="64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa"
Oct 31 07:47:53.187481 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.187442    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa"} err="failed to get container status \"64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa\": rpc error: code = NotFound desc = could not find container \"64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa\": container with ID starting with 64db9028705f041c5cb164eb50fafaa7f8d6253c6bb2bb4555b2236f8d4feffa not found: ID does not exist"
Oct 31 07:47:53.193697 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.193675    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"]
Oct 31 07:47:53.199024 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.199004    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-75d6970a-predictor-8569ddb866-wzzt4"]
Oct 31 07:47:53.419197 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.419164    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" path="/var/lib/kubelet/pods/b8ed6857-c101-48a8-8bff-6bcb2706fe44/volumes"
Oct 31 07:47:53.419551 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:53.419537    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef762d1b-c67f-44b7-ab32-5a3df91f13c5" path="/var/lib/kubelet/pods/ef762d1b-c67f-44b7-ab32-5a3df91f13c5/volumes"
Oct 31 07:47:55.361541 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361511    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"]
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361814    4913 cpu_manager.go:414] "containerMap: removing container" podUID="ef762d1b-c67f-44b7-ab32-5a3df91f13c5" containerName="kserve-container"
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361826    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef762d1b-c67f-44b7-ab32-5a3df91f13c5" containerName="kserve-container"
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361856    4913 cpu_manager.go:414] "containerMap: removing container" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy"
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361860    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy"
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361873    4913 cpu_manager.go:414] "containerMap: removing container" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kserve-container"
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361878    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kserve-container"
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361929    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kube-rbac-proxy"
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361940    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ed6857-c101-48a8-8bff-6bcb2706fe44" containerName="kserve-container"
Oct 31 07:47:55.361950 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.361946    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef762d1b-c67f-44b7-ab32-5a3df91f13c5" containerName="kserve-container"
Oct 31 07:47:55.366608 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.366580    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:47:55.374156 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.374133    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:47:55.374361 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.374341    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:47:55.376397 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.374436    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:47:55.376397 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.376174    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"]
Oct 31 07:47:55.549255 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.549216    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnd6p\" (UniqueName: \"kubernetes.io/projected/a5988a1f-1159-4584-9067-2c0022c98925-kube-api-access-nnd6p\") pod \"proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47\" (UID: \"a5988a1f-1159-4584-9067-2c0022c98925\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:47:55.650350 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.650256    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd6p\" (UniqueName: \"kubernetes.io/projected/a5988a1f-1159-4584-9067-2c0022c98925-kube-api-access-nnd6p\") pod \"proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47\" (UID: \"a5988a1f-1159-4584-9067-2c0022c98925\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:47:55.658912 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.658881    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd6p\" (UniqueName: \"kubernetes.io/projected/a5988a1f-1159-4584-9067-2c0022c98925-kube-api-access-nnd6p\") pod \"proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47\" (UID: \"a5988a1f-1159-4584-9067-2c0022c98925\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:47:55.680727 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.680702    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:47:55.952491 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:55.952458    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"]
Oct 31 07:47:55.955951 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:47:55.955916    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5988a1f_1159_4584_9067_2c0022c98925.slice/crio-e3494a84a06d28972eb0e5f314378e21df22962e7749a764761327f44116b390 WatchSource:0}: Error finding container e3494a84a06d28972eb0e5f314378e21df22962e7749a764761327f44116b390: Status 404 returned error can't find the container with id e3494a84a06d28972eb0e5f314378e21df22962e7749a764761327f44116b390
Oct 31 07:47:56.183983 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:56.183946    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47" event={"ID":"a5988a1f-1159-4584-9067-2c0022c98925","Type":"ContainerStarted","Data":"d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12"}
Oct 31 07:47:56.184195 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:56.184034    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47" event={"ID":"a5988a1f-1159-4584-9067-2c0022c98925","Type":"ContainerStarted","Data":"e3494a84a06d28972eb0e5f314378e21df22962e7749a764761327f44116b390"}
Oct 31 07:47:56.184195 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:56.184056    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:47:56.185755 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:56.185732    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47" podUID="a5988a1f-1159-4584-9067-2c0022c98925" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/\": dial tcp 10.132.0.46:8080: connect: connection refused"
Oct 31 07:47:56.198510 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:56.198458    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47" podStartSLOduration=1.198445005 podStartE2EDuration="1.198445005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:47:56.19812104 +0000 UTC m=+1124.887889359" watchObservedRunningTime="2026-10-31 07:47:56.198445005 +0000 UTC m=+1124.888213297"
Oct 31 07:47:57.189035 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:57.189004    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:47:58.961034 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:47:58.960994    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 31 07:48:00.684736 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.684699    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"]
Oct 31 07:48:00.688414 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.688393    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.691261 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.691238    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-ab6c87fb-kube-rbac-proxy-sar-config\""
Oct 31 07:48:00.692030 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.692009    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-ab6c87fb-predictor-serving-cert\""
Oct 31 07:48:00.692129 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.692044    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-ab6c87fb-dockercfg-zcgf8\""
Oct 31 07:48:00.697526 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.697507    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"]
Oct 31 07:48:00.788063 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.788026    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082fe3c2-c3e1-4786-89ab-e05526880e9b-proxy-tls\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.788264 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.788077    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxpp\" (UniqueName: \"kubernetes.io/projected/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-api-access-skxpp\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.788313 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.788258    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.889615 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.889573    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.889839 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.889628    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082fe3c2-c3e1-4786-89ab-e05526880e9b-proxy-tls\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.889839 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.889659    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxpp\" (UniqueName: \"kubernetes.io/projected/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-api-access-skxpp\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.890348 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.890322    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-rbac-proxy-sar-config\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.892056 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.892034    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082fe3c2-c3e1-4786-89ab-e05526880e9b-proxy-tls\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:00.897382 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:00.897363    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxpp\" (UniqueName: \"kubernetes.io/projected/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-api-access-skxpp\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.409296 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.409254    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.415103 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.415074    4913 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") device path: \"\"" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.510227 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.510175    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.510429 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.510331    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") DevicePath \"\"" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.513136 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.513113    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") DevicePath \"csi-8a85d992598a057667202e54751df3cb2ad8ad45e1338f120085045e4521f6eb\"" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.678280 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.678241    4913 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d8cb902dceb2962e65c6d76a3aecf3fda133d2396db606dc98ccff0dc0325859/globalmount\"" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.691507 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.691480    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") pod \"proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") " pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.705209 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.705175    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:03.983296 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:03.983204    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"]
Oct 31 07:48:03.986307 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:48:03.986274    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082fe3c2_c3e1_4786_89ab_e05526880e9b.slice/crio-b9f1dd8bc190fa68005f10d21a731c9908384ff53602b135d000ae205bc2679a WatchSource:0}: Error finding container b9f1dd8bc190fa68005f10d21a731c9908384ff53602b135d000ae205bc2679a: Status 404 returned error can't find the container with id b9f1dd8bc190fa68005f10d21a731c9908384ff53602b135d000ae205bc2679a
Oct 31 07:48:04.222000 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:04.221961    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" event={"ID":"082fe3c2-c3e1-4786-89ab-e05526880e9b","Type":"ContainerStarted","Data":"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c"}
Oct 31 07:48:04.222222 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:04.222028    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" event={"ID":"082fe3c2-c3e1-4786-89ab-e05526880e9b","Type":"ContainerStarted","Data":"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c"}
Oct 31 07:48:04.222222 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:04.222061    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" event={"ID":"082fe3c2-c3e1-4786-89ab-e05526880e9b","Type":"ContainerStarted","Data":"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615"}
Oct 31 07:48:04.222356 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:04.222282    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:04.222356 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:04.222343    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" event={"ID":"082fe3c2-c3e1-4786-89ab-e05526880e9b","Type":"ContainerStarted","Data":"b9f1dd8bc190fa68005f10d21a731c9908384ff53602b135d000ae205bc2679a"}
Oct 31 07:48:04.222462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:04.222388    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:04.223693 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:04.223659    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.47:8080/\": dial tcp 10.132.0.47:8080: connect: connection refused"
Oct 31 07:48:04.240199 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:04.240078    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podStartSLOduration=7.240061745 podStartE2EDuration="7.240061745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:48:04.239938858 +0000 UTC m=+1132.929707151" watchObservedRunningTime="2026-10-31 07:48:04.240061745 +0000 UTC m=+1132.929830034"
Oct 31 07:48:05.226521 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:05.226481    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:05.227827 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:05.227805    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:05.228266 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:05.228248    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:09.522523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:09.522483    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:48:12.242356 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:12.242326    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:48:16.057748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:16.057717    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"]
Oct 31 07:48:16.058161 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:16.057973    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kserve-container" containerID="cri-o://3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba" gracePeriod=30
Oct 31 07:48:16.058161 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:16.058046    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kube-rbac-proxy" containerID="cri-o://9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b" gracePeriod=30
Oct 31 07:48:16.279207 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:16.279154    4913 generic.go:356] "Generic (PLEG): container finished" podID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerID="9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b" exitCode=0
Oct 31 07:48:16.279367 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:16.279219    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" event={"ID":"6250c4c7-c6fc-47af-bfaf-693020fb62c6","Type":"ContainerDied","Data":"9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b"}
Oct 31 07:48:18.954716 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:18.954676    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Oct 31 07:48:18.961171 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:18.961138    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Oct 31 07:48:19.399857 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:48:19.399773    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:48:21.006864 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.006839    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:48:21.051829 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.051796    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kserve-provision-location\") pod \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") "
Oct 31 07:48:21.051995 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.051851    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6250c4c7-c6fc-47af-bfaf-693020fb62c6-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6250c4c7-c6fc-47af-bfaf-693020fb62c6-proxy-tls\") pod \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") "
Oct 31 07:48:21.051995 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.051882    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-api-access-89vb8\" (UniqueName: \"kubernetes.io/projected/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-api-access-89vb8\") pod \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") "
Oct 31 07:48:21.051995 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.051916    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-rbac-proxy-sar-config\") pod \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") "
Oct 31 07:48:21.052136 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.052115    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kserve-provision-location" pod "6250c4c7-c6fc-47af-bfaf-693020fb62c6" (UID: "6250c4c7-c6fc-47af-bfaf-693020fb62c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:48:21.052333 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.052309    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-rbac-proxy-sar-config" pod "6250c4c7-c6fc-47af-bfaf-693020fb62c6" (UID: "6250c4c7-c6fc-47af-bfaf-693020fb62c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:48:21.054016 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.053988    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6250c4c7-c6fc-47af-bfaf-693020fb62c6-proxy-tls" pod "6250c4c7-c6fc-47af-bfaf-693020fb62c6" (UID: "6250c4c7-c6fc-47af-bfaf-693020fb62c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:48:21.054129 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.054049    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-api-access-89vb8" pod "6250c4c7-c6fc-47af-bfaf-693020fb62c6" (UID: "6250c4c7-c6fc-47af-bfaf-693020fb62c6"). InnerVolumeSpecName "kube-api-access-89vb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:48:21.153275 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.153193    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-cabundle-cert\") pod \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\" (UID: \"6250c4c7-c6fc-47af-bfaf-693020fb62c6\") "
Oct 31 07:48:21.153422 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.153356    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:48:21.153422 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.153367    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6250c4c7-c6fc-47af-bfaf-693020fb62c6-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:48:21.153422 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.153378    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89vb8\" (UniqueName: \"kubernetes.io/projected/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-api-access-89vb8\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:48:21.153422 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.153387    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:48:21.153559 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.153527    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-cabundle-cert" pod "6250c4c7-c6fc-47af-bfaf-693020fb62c6" (UID: "6250c4c7-c6fc-47af-bfaf-693020fb62c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:48:21.253771 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.253729    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6250c4c7-c6fc-47af-bfaf-693020fb62c6-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:48:21.301377 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.301342    4913 generic.go:356] "Generic (PLEG): container finished" podID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerID="3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba" exitCode=0
Oct 31 07:48:21.301576 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.301425    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"
Oct 31 07:48:21.301576 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.301424    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" event={"ID":"6250c4c7-c6fc-47af-bfaf-693020fb62c6","Type":"ContainerDied","Data":"3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba"}
Oct 31 07:48:21.301576 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.301534    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc" event={"ID":"6250c4c7-c6fc-47af-bfaf-693020fb62c6","Type":"ContainerDied","Data":"b8bd5be00c8e0a4ab3a0e49f67cf40542e6f6a64ba0fb6eca5e168d945bfb0b7"}
Oct 31 07:48:21.301576 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.301573    4913 scope.go:122] "RemoveContainer" containerID="9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b"
Oct 31 07:48:21.310315 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.310291    4913 scope.go:122] "RemoveContainer" containerID="3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba"
Oct 31 07:48:21.317973 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.317954    4913 scope.go:122] "RemoveContainer" containerID="93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630"
Oct 31 07:48:21.323434 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.323410    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"]
Oct 31 07:48:21.327422 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.327399    4913 scope.go:122] "RemoveContainer" containerID="9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b"
Oct 31 07:48:21.328496 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:48:21.328469    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b\": container with ID starting with 9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b not found: ID does not exist" containerID="9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b"
Oct 31 07:48:21.328604 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.328516    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b"} err="failed to get container status \"9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b\": rpc error: code = NotFound desc = could not find container \"9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b\": container with ID starting with 9421ebcb148d79a08305ec4c0684ae7ce50e4a83b3e2c07c12cc56aa5cd0644b not found: ID does not exist"
Oct 31 07:48:21.328604 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.328541    4913 scope.go:122] "RemoveContainer" containerID="3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba"
Oct 31 07:48:21.329256 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:48:21.329232    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba\": container with ID starting with 3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba not found: ID does not exist" containerID="3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba"
Oct 31 07:48:21.329352 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.329265    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba"} err="failed to get container status \"3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba\": rpc error: code = NotFound desc = could not find container \"3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba\": container with ID starting with 3bb3001d3274dd23177a1bc5404aa526338062a8408513a8be0e6e18fd211eba not found: ID does not exist"
Oct 31 07:48:21.329352 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.329288    4913 scope.go:122] "RemoveContainer" containerID="93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630"
Oct 31 07:48:21.329352 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.329326    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-predictive-sklearn-v2-predictor-6c6fcd499d-zvkdc"]
Oct 31 07:48:21.329676 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:48:21.329651    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630\": container with ID starting with 93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630 not found: ID does not exist" containerID="93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630"
Oct 31 07:48:21.329733 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.329686    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630"} err="failed to get container status \"93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630\": rpc error: code = NotFound desc = could not find container \"93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630\": container with ID starting with 93778516b03aad67b8ba4c2b535577bbe57c68b28bfa2f8412159e4368992630 not found: ID does not exist"
Oct 31 07:48:21.420156 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:21.420079    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" path="/var/lib/kubelet/pods/6250c4c7-c6fc-47af-bfaf-693020fb62c6/volumes"
Oct 31 07:48:48.215944 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:48.215904    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"]
Oct 31 07:48:48.218480 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:48.216147    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47" podUID="a5988a1f-1159-4584-9067-2c0022c98925" containerName="kserve-container" containerID="cri-o://d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12" gracePeriod=30
Oct 31 07:48:48.292339 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:48.292301    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"]
Oct 31 07:48:48.292682 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:48.292643    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kserve-container" containerID="cri-o://c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615" gracePeriod=30
Oct 31 07:48:48.293078 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:48.293017    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="agent" containerID="cri-o://57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c" gracePeriod=30
Oct 31 07:48:48.293218 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:48.293062    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy" containerID="cri-o://fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c" gracePeriod=30
Oct 31 07:48:48.417263 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:48.417234    4913 generic.go:356] "Generic (PLEG): container finished" podID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerID="fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c" exitCode=0
Oct 31 07:48:48.417396 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:48.417288    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" event={"ID":"082fe3c2-c3e1-4786-89ab-e05526880e9b","Type":"ContainerDied","Data":"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c"}
Oct 31 07:48:49.400398 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:48:49.400363    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:48:52.238272 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:52.238224    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 31 07:48:55.226932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:55.226896    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:48:57.238387 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:48:57.238345    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 31 07:49:02.238358 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:02.238313    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 31 07:49:02.238782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:02.238460    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:49:05.228024 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:05.227975    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:49:07.237888 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:07.237841    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 31 07:49:11.515799 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:11.515772    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:49:11.518928 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:11.518907    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:49:12.237959 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:12.237920    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 31 07:49:15.227943 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:15.227901    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:49:15.228459 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:15.228052    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:49:17.238316 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:17.238271    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Oct 31 07:49:18.410433 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.410107    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:49:18.480220 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.480195    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:49:18.517838 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.517806    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5988a1f-1159-4584-9067-2c0022c98925-kube-api-access-nnd6p\" (UniqueName: \"kubernetes.io/projected/a5988a1f-1159-4584-9067-2c0022c98925-kube-api-access-nnd6p\") pod \"a5988a1f-1159-4584-9067-2c0022c98925\" (UID: \"a5988a1f-1159-4584-9067-2c0022c98925\") "
Oct 31 07:49:18.519779 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.519751    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5988a1f-1159-4584-9067-2c0022c98925-kube-api-access-nnd6p" pod "a5988a1f-1159-4584-9067-2c0022c98925" (UID: "a5988a1f-1159-4584-9067-2c0022c98925"). InnerVolumeSpecName "kube-api-access-nnd6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:49:18.542604 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.542572    4913 generic.go:356] "Generic (PLEG): container finished" podID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerID="57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c" exitCode=0
Oct 31 07:49:18.542604 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.542601    4913 generic.go:356] "Generic (PLEG): container finished" podID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerID="c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615" exitCode=137
Oct 31 07:49:18.542777 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.542637    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"
Oct 31 07:49:18.542777 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.542648    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" event={"ID":"082fe3c2-c3e1-4786-89ab-e05526880e9b","Type":"ContainerDied","Data":"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c"}
Oct 31 07:49:18.542777 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.542703    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" event={"ID":"082fe3c2-c3e1-4786-89ab-e05526880e9b","Type":"ContainerDied","Data":"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615"}
Oct 31 07:49:18.542777 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.542731    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv" event={"ID":"082fe3c2-c3e1-4786-89ab-e05526880e9b","Type":"ContainerDied","Data":"b9f1dd8bc190fa68005f10d21a731c9908384ff53602b135d000ae205bc2679a"}
Oct 31 07:49:18.542777 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.542771    4913 scope.go:122] "RemoveContainer" containerID="57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c"
Oct 31 07:49:18.543908 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.543888    4913 generic.go:356] "Generic (PLEG): container finished" podID="a5988a1f-1159-4584-9067-2c0022c98925" containerID="d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12" exitCode=137
Oct 31 07:49:18.544004 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.543919    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"
Oct 31 07:49:18.544004 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.543942    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47" event={"ID":"a5988a1f-1159-4584-9067-2c0022c98925","Type":"ContainerDied","Data":"d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12"}
Oct 31 07:49:18.544004 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.543973    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47" event={"ID":"a5988a1f-1159-4584-9067-2c0022c98925","Type":"ContainerDied","Data":"e3494a84a06d28972eb0e5f314378e21df22962e7749a764761327f44116b390"}
Oct 31 07:49:18.550992 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.550978    4913 scope.go:122] "RemoveContainer" containerID="fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c"
Oct 31 07:49:18.558543 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.558523    4913 scope.go:122] "RemoveContainer" containerID="c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615"
Oct 31 07:49:18.563557 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.563534    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"]
Oct 31 07:49:18.565765 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.565744    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-sink-78c8cf8fd-hsd47"]
Oct 31 07:49:18.566414 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.566394    4913 scope.go:122] "RemoveContainer" containerID="57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c"
Oct 31 07:49:18.566676 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:49:18.566658    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c\": container with ID starting with 57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c not found: ID does not exist" containerID="57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c"
Oct 31 07:49:18.566756 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.566683    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c"} err="failed to get container status \"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c\": rpc error: code = NotFound desc = could not find container \"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c\": container with ID starting with 57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c not found: ID does not exist"
Oct 31 07:49:18.566756 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.566700    4913 scope.go:122] "RemoveContainer" containerID="fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c"
Oct 31 07:49:18.566934 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:49:18.566917    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c\": container with ID starting with fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c not found: ID does not exist" containerID="fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c"
Oct 31 07:49:18.566977 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.566942    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c"} err="failed to get container status \"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c\": rpc error: code = NotFound desc = could not find container \"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c\": container with ID starting with fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c not found: ID does not exist"
Oct 31 07:49:18.566977 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.566960    4913 scope.go:122] "RemoveContainer" containerID="c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615"
Oct 31 07:49:18.567180 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:49:18.567166    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615\": container with ID starting with c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615 not found: ID does not exist" containerID="c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615"
Oct 31 07:49:18.567180 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.567195    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615"} err="failed to get container status \"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615\": rpc error: code = NotFound desc = could not find container \"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615\": container with ID starting with c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615 not found: ID does not exist"
Oct 31 07:49:18.567312 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.567207    4913 scope.go:122] "RemoveContainer" containerID="57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c"
Oct 31 07:49:18.567444 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.567426    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c"} err="failed to get container status \"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c\": rpc error: code = NotFound desc = could not find container \"57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c\": container with ID starting with 57a724ae7d0253864634f5cf010fff831bf721ef50288e482b479744cf597f2c not found: ID does not exist"
Oct 31 07:49:18.567484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.567444    4913 scope.go:122] "RemoveContainer" containerID="fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c"
Oct 31 07:49:18.567662 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.567642    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c"} err="failed to get container status \"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c\": rpc error: code = NotFound desc = could not find container \"fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c\": container with ID starting with fe280d1ec3e7af45affde2f7d449600ad3f4d77610402da94e90aa5260d5fd5c not found: ID does not exist"
Oct 31 07:49:18.567708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.567662    4913 scope.go:122] "RemoveContainer" containerID="c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615"
Oct 31 07:49:18.567905 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.567887    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615"} err="failed to get container status \"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615\": rpc error: code = NotFound desc = could not find container \"c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615\": container with ID starting with c33a5591a5e8b57468d5875d36856a7dd0ae57022101c71d1eb5f58232163615 not found: ID does not exist"
Oct 31 07:49:18.567946 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.567905    4913 scope.go:122] "RemoveContainer" containerID="d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12"
Oct 31 07:49:18.574890 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.574872    4913 scope.go:122] "RemoveContainer" containerID="d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12"
Oct 31 07:49:18.575146 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:49:18.575129    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12\": container with ID starting with d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12 not found: ID does not exist" containerID="d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12"
Oct 31 07:49:18.575213 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.575150    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12"} err="failed to get container status \"d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12\": rpc error: code = NotFound desc = could not find container \"d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12\": container with ID starting with d4865c193be60f0571d9c7a15e37aac9f42f073d265be2432d9d17feee4caa12 not found: ID does not exist"
Oct 31 07:49:18.618360 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.618333    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") pod \"082fe3c2-c3e1-4786-89ab-e05526880e9b\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") "
Oct 31 07:49:18.618501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.618385    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/082fe3c2-c3e1-4786-89ab-e05526880e9b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/082fe3c2-c3e1-4786-89ab-e05526880e9b-proxy-tls\") pod \"082fe3c2-c3e1-4786-89ab-e05526880e9b\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") "
Oct 31 07:49:18.618501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.618455    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-api-access-skxpp\" (UniqueName: \"kubernetes.io/projected/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-api-access-skxpp\") pod \"082fe3c2-c3e1-4786-89ab-e05526880e9b\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") "
Oct 31 07:49:18.618501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.618492    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-rbac-proxy-sar-config\") pod \"082fe3c2-c3e1-4786-89ab-e05526880e9b\" (UID: \"082fe3c2-c3e1-4786-89ab-e05526880e9b\") "
Oct 31 07:49:18.618720 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.618700    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnd6p\" (UniqueName: \"kubernetes.io/projected/a5988a1f-1159-4584-9067-2c0022c98925-kube-api-access-nnd6p\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:18.618910 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.618874    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-rbac-proxy-sar-config" pod "082fe3c2-c3e1-4786-89ab-e05526880e9b" (UID: "082fe3c2-c3e1-4786-89ab-e05526880e9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:49:18.620549 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.620522    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-api-access-skxpp" pod "082fe3c2-c3e1-4786-89ab-e05526880e9b" (UID: "082fe3c2-c3e1-4786-89ab-e05526880e9b"). InnerVolumeSpecName "kube-api-access-skxpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:49:18.620652 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.620580    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082fe3c2-c3e1-4786-89ab-e05526880e9b-proxy-tls" pod "082fe3c2-c3e1-4786-89ab-e05526880e9b" (UID: "082fe3c2-c3e1-4786-89ab-e05526880e9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:49:18.620894 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.620878    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87" pod "082fe3c2-c3e1-4786-89ab-e05526880e9b" (UID: "082fe3c2-c3e1-4786-89ab-e05526880e9b"). InnerVolumeSpecName "pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:49:18.719923 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.719887    4913 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" "
Oct 31 07:49:18.719923 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.719919    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082fe3c2-c3e1-4786-89ab-e05526880e9b-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:18.719923 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.719930    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxpp\" (UniqueName: \"kubernetes.io/projected/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-api-access-skxpp\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:18.720138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.719940    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082fe3c2-c3e1-4786-89ab-e05526880e9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:49:18.729763 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.729743    4913 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87") on node "ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:49:18.821083 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.821043    4913 reconciler_common.go:299] "Volume detached for volume \"pvc-0bdb58b0-816f-4a29-b5d5-45de89a00f0e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d349fb3fdeacaf87\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"csi-8a85d992598a057667202e54751df3cb2ad8ad45e1338f120085045e4521f6eb\""
Oct 31 07:49:18.863691 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.863660    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"]
Oct 31 07:49:18.867048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:18.867023    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-ab6c87fb-predictor-7896d7754d-wtdfv"]
Oct 31 07:49:19.193655 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.193617    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"]
Oct 31 07:49:19.194023 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194005    4913 cpu_manager.go:414] "containerMap: removing container" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="agent"
Oct 31 07:49:19.194023 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194024    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="agent"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194046    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kserve-container"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194052    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kserve-container"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194062    4913 cpu_manager.go:414] "containerMap: removing container" podUID="a5988a1f-1159-4584-9067-2c0022c98925" containerName="kserve-container"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194067    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5988a1f-1159-4584-9067-2c0022c98925" containerName="kserve-container"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194077    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kube-rbac-proxy"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194082    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kube-rbac-proxy"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194089    4913 cpu_manager.go:414] "containerMap: removing container" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kserve-container"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194094    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kserve-container"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194102    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="storage-initializer"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194108    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="storage-initializer"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194118    4913 cpu_manager.go:414] "containerMap: removing container" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194123    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194169    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kserve-container"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194178    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5988a1f-1159-4584-9067-2c0022c98925" containerName="kserve-container"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194203    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6250c4c7-c6fc-47af-bfaf-693020fb62c6" containerName="kube-rbac-proxy"
Oct 31 07:49:19.194210 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194211    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kserve-container"
Oct 31 07:49:19.194660 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194221    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="kube-rbac-proxy"
Oct 31 07:49:19.194660 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.194230    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" containerName="agent"
Oct 31 07:49:19.198575 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.198552    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:49:19.203938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.203911    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"]
Oct 31 07:49:19.207234 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.205808    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:49:19.207234 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.205914    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:49:19.208094 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.208074    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:49:19.325822 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.325788    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc4k5\" (UniqueName: \"kubernetes.io/projected/6b08558a-0c91-415d-82b9-2ea0d3d43a5e-kube-api-access-jc4k5\") pod \"proxy-routing-e5465674-sink-884fd44d9-brfck\" (UID: \"6b08558a-0c91-415d-82b9-2ea0d3d43a5e\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:49:19.399916 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:49:19.399879    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:49:19.419217 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.419160    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082fe3c2-c3e1-4786-89ab-e05526880e9b" path="/var/lib/kubelet/pods/082fe3c2-c3e1-4786-89ab-e05526880e9b/volumes"
Oct 31 07:49:19.419797 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.419775    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5988a1f-1159-4584-9067-2c0022c98925" path="/var/lib/kubelet/pods/a5988a1f-1159-4584-9067-2c0022c98925/volumes"
Oct 31 07:49:19.426513 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.426494    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4k5\" (UniqueName: \"kubernetes.io/projected/6b08558a-0c91-415d-82b9-2ea0d3d43a5e-kube-api-access-jc4k5\") pod \"proxy-routing-e5465674-sink-884fd44d9-brfck\" (UID: \"6b08558a-0c91-415d-82b9-2ea0d3d43a5e\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:49:19.434662 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.434637    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4k5\" (UniqueName: \"kubernetes.io/projected/6b08558a-0c91-415d-82b9-2ea0d3d43a5e-kube-api-access-jc4k5\") pod \"proxy-routing-e5465674-sink-884fd44d9-brfck\" (UID: \"6b08558a-0c91-415d-82b9-2ea0d3d43a5e\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:49:19.513245 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.513161    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:49:19.783352 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:19.783273    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"]
Oct 31 07:49:20.555523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:20.555483    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck" event={"ID":"6b08558a-0c91-415d-82b9-2ea0d3d43a5e","Type":"ContainerStarted","Data":"a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3"}
Oct 31 07:49:20.555965 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:20.555574    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck" event={"ID":"6b08558a-0c91-415d-82b9-2ea0d3d43a5e","Type":"ContainerStarted","Data":"cbe1457f538e7659c36d85355c93293abeb1aa9a11e2a5821306f1b2816951ad"}
Oct 31 07:49:20.555965 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:20.555605    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:49:20.558068 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:20.558044    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:49:20.570521 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:20.570478    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck" podStartSLOduration=1.5704664510000002 podStartE2EDuration="1.570466451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:49:20.569239409 +0000 UTC m=+1209.259007701" watchObservedRunningTime="2026-10-31 07:49:20.570466451 +0000 UTC m=+1209.260234740"
Oct 31 07:49:24.557446 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.557413    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"]
Oct 31 07:49:24.560925 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.560908    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.564117 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.564092    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-e5465674-kube-rbac-proxy-sar-config\""
Oct 31 07:49:24.565014 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.564993    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-e5465674-predictor-serving-cert\""
Oct 31 07:49:24.565014 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.564999    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c94084-6f7e-4bf1-915d-1d0459b00edb-proxy-tls\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.565133 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.564998    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-e5465674-dockercfg-hkhdq\""
Oct 31 07:49:24.565133 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.565074    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.565133 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.565119    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjdx8\" (UniqueName: \"kubernetes.io/projected/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-api-access-qjdx8\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.569859 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.569837    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"]
Oct 31 07:49:24.666368 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.666328    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.666567 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.666422    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjdx8\" (UniqueName: \"kubernetes.io/projected/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-api-access-qjdx8\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.666567 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.666472    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c94084-6f7e-4bf1-915d-1d0459b00edb-proxy-tls\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.667147 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.667123    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.669052 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.669025    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c94084-6f7e-4bf1-915d-1d0459b00edb-proxy-tls\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:24.673942 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:24.673913    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjdx8\" (UniqueName: \"kubernetes.io/projected/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-api-access-qjdx8\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.005609 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.005563    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.011811 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.011782    4913 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") device path: \"\"" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.106463 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.106419    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.106634 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.106557    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") DevicePath \"\"" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.109463 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.109442    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") DevicePath \"csi-6b020105ec0ae3d8191bba9389100371262bb677ec41338209cf16e393dfb259\"" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.274996 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.274919    4913 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b209097d5069d9167df07ac627b6dfcd91d6e47fdf236aa8fefb9a8a95a6d75c/globalmount\"" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.287640 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.287613    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.377394 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.377365    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:29.650397 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:29.650373    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"]
Oct 31 07:49:29.652949 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:49:29.652920    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c94084_6f7e_4bf1_915d_1d0459b00edb.slice/crio-9f645a3c799c435810c4d43b966deaf77373a01093a2b1ca025af87a2e8a0b2e WatchSource:0}: Error finding container 9f645a3c799c435810c4d43b966deaf77373a01093a2b1ca025af87a2e8a0b2e: Status 404 returned error can't find the container with id 9f645a3c799c435810c4d43b966deaf77373a01093a2b1ca025af87a2e8a0b2e
Oct 31 07:49:30.602537 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:30.602502    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" event={"ID":"89c94084-6f7e-4bf1-915d-1d0459b00edb","Type":"ContainerStarted","Data":"0200bb7cc46c7f260c5d6ec2394849f3d32cfd42f9df662f9e716478e45b5411"}
Oct 31 07:49:30.602999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:30.602558    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" event={"ID":"89c94084-6f7e-4bf1-915d-1d0459b00edb","Type":"ContainerStarted","Data":"eb2b45ac4ae24c1ecc03d3981512b1b240f3d4c08f3c7aece0a8b9bf0fb990af"}
Oct 31 07:49:30.602999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:30.602578    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" event={"ID":"89c94084-6f7e-4bf1-915d-1d0459b00edb","Type":"ContainerStarted","Data":"9f645a3c799c435810c4d43b966deaf77373a01093a2b1ca025af87a2e8a0b2e"}
Oct 31 07:49:30.602999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:30.602707    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:30.602999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:30.602758    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:30.605116 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:30.605096    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:30.621777 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:30.621722    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podStartSLOduration=9.621708692 podStartE2EDuration="9.621708692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:49:30.619489386 +0000 UTC m=+1219.309257677" watchObservedRunningTime="2026-10-31 07:49:30.621708692 +0000 UTC m=+1219.311476985"
Oct 31 07:49:37.615227 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:37.615179    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:49:41.654692 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.654655    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"]
Oct 31 07:49:41.658326 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.658305    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.667456 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.667432    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"]
Oct 31 07:49:41.810247 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.810214    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e03291a-ede6-438c-a841-2248aa463f0c-proxy-tls\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.810419 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.810259    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.810419 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.810380    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e03291a-ede6-438c-a841-2248aa463f0c-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.810419 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.810380    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") DevicePath \"csi-6b020105ec0ae3d8191bba9389100371262bb677ec41338209cf16e393dfb259\"" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.810590 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.810427    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hqs\" (UniqueName: \"kubernetes.io/projected/7e03291a-ede6-438c-a841-2248aa463f0c-kube-api-access-r6hqs\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.813561 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.813540    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") DevicePath \"csi-6b020105ec0ae3d8191bba9389100371262bb677ec41338209cf16e393dfb259\"" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.824853 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.824828    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.910984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.910895    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e03291a-ede6-438c-a841-2248aa463f0c-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.910984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.910936    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hqs\" (UniqueName: \"kubernetes.io/projected/7e03291a-ede6-438c-a841-2248aa463f0c-kube-api-access-r6hqs\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.911223 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.911073    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e03291a-ede6-438c-a841-2248aa463f0c-proxy-tls\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.911609 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.911588    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e03291a-ede6-438c-a841-2248aa463f0c-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.913514 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.913489    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e03291a-ede6-438c-a841-2248aa463f0c-proxy-tls\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.919154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.919132    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hqs\" (UniqueName: \"kubernetes.io/projected/7e03291a-ede6-438c-a841-2248aa463f0c-kube-api-access-r6hqs\") pod \"proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") " pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:41.973084 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:41.973049    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:42.248412 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:42.248315    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"]
Oct 31 07:49:42.251014 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:49:42.250985    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e03291a_ede6_438c_a841_2248aa463f0c.slice/crio-1aec602339aa7cb568af62d44f359c47dd11f5f126e5598ad9e952228bfe55df WatchSource:0}: Error finding container 1aec602339aa7cb568af62d44f359c47dd11f5f126e5598ad9e952228bfe55df: Status 404 returned error can't find the container with id 1aec602339aa7cb568af62d44f359c47dd11f5f126e5598ad9e952228bfe55df
Oct 31 07:49:42.654248 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:42.654128    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" event={"ID":"7e03291a-ede6-438c-a841-2248aa463f0c","Type":"ContainerStarted","Data":"cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f"}
Oct 31 07:49:42.654248 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:42.654227    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" event={"ID":"7e03291a-ede6-438c-a841-2248aa463f0c","Type":"ContainerStarted","Data":"97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f"}
Oct 31 07:49:42.654435 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:42.654261    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" event={"ID":"7e03291a-ede6-438c-a841-2248aa463f0c","Type":"ContainerStarted","Data":"1aec602339aa7cb568af62d44f359c47dd11f5f126e5598ad9e952228bfe55df"}
Oct 31 07:49:42.654435 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:42.654422    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:42.672165 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:42.672118    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podStartSLOduration=1.672105217 podStartE2EDuration="1.672105217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:49:42.671419859 +0000 UTC m=+1231.361188169" watchObservedRunningTime="2026-10-31 07:49:42.672105217 +0000 UTC m=+1231.361873510"
Oct 31 07:49:43.658613 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:43.658583    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:43.660678 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:43.660657    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:49.399381 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:49:49.399340    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:49:50.670610 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:50.670577    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:49:50.712527 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:50.712497    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"]
Oct 31 07:49:50.712774 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:50.712747    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kserve-container" containerID="cri-o://eb2b45ac4ae24c1ecc03d3981512b1b240f3d4c08f3c7aece0a8b9bf0fb990af" gracePeriod=30
Oct 31 07:49:50.712851 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:50.712795    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy" containerID="cri-o://0200bb7cc46c7f260c5d6ec2394849f3d32cfd42f9df662f9e716478e45b5411" gracePeriod=30
Oct 31 07:49:51.692902 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:51.692867    4913 generic.go:356] "Generic (PLEG): container finished" podID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerID="0200bb7cc46c7f260c5d6ec2394849f3d32cfd42f9df662f9e716478e45b5411" exitCode=0
Oct 31 07:49:51.693318 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:51.692938    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" event={"ID":"89c94084-6f7e-4bf1-915d-1d0459b00edb","Type":"ContainerDied","Data":"0200bb7cc46c7f260c5d6ec2394849f3d32cfd42f9df662f9e716478e45b5411"}
Oct 31 07:49:52.611812 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:52.611770    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Oct 31 07:49:57.611644 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:49:57.611600    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Oct 31 07:50:02.611298 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:02.611249    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Oct 31 07:50:02.611708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:02.611359    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:50:07.611418 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:07.611377    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Oct 31 07:50:12.612031 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:12.611986    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Oct 31 07:50:17.612154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:17.612112    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Oct 31 07:50:19.400307 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:50:19.400269    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:50:20.789238 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:50:20.789203    4913 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c94084_6f7e_4bf1_915d_1d0459b00edb.slice/crio-conmon-eb2b45ac4ae24c1ecc03d3981512b1b240f3d4c08f3c7aece0a8b9bf0fb990af.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 07:50:20.813251 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.813219    4913 generic.go:356] "Generic (PLEG): container finished" podID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerID="eb2b45ac4ae24c1ecc03d3981512b1b240f3d4c08f3c7aece0a8b9bf0fb990af" exitCode=137
Oct 31 07:50:20.813438 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.813291    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" event={"ID":"89c94084-6f7e-4bf1-915d-1d0459b00edb","Type":"ContainerDied","Data":"eb2b45ac4ae24c1ecc03d3981512b1b240f3d4c08f3c7aece0a8b9bf0fb990af"}
Oct 31 07:50:20.901673 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.901650    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:50:20.954381 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.954345    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"89c94084-6f7e-4bf1-915d-1d0459b00edb\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") "
Oct 31 07:50:20.954559 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.954416    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-api-access-qjdx8\" (UniqueName: \"kubernetes.io/projected/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-api-access-qjdx8\") pod \"89c94084-6f7e-4bf1-915d-1d0459b00edb\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") "
Oct 31 07:50:20.954559 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.954462    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-rbac-proxy-sar-config\") pod \"89c94084-6f7e-4bf1-915d-1d0459b00edb\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") "
Oct 31 07:50:20.954559 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.954504    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c94084-6f7e-4bf1-915d-1d0459b00edb-proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c94084-6f7e-4bf1-915d-1d0459b00edb-proxy-tls\") pod \"89c94084-6f7e-4bf1-915d-1d0459b00edb\" (UID: \"89c94084-6f7e-4bf1-915d-1d0459b00edb\") "
Oct 31 07:50:20.954868 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.954837    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-rbac-proxy-sar-config" pod "89c94084-6f7e-4bf1-915d-1d0459b00edb" (UID: "89c94084-6f7e-4bf1-915d-1d0459b00edb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:50:20.956755 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.956730    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-api-access-qjdx8" pod "89c94084-6f7e-4bf1-915d-1d0459b00edb" (UID: "89c94084-6f7e-4bf1-915d-1d0459b00edb"). InnerVolumeSpecName "kube-api-access-qjdx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:50:20.956856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.956754    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c94084-6f7e-4bf1-915d-1d0459b00edb-proxy-tls" pod "89c94084-6f7e-4bf1-915d-1d0459b00edb" (UID: "89c94084-6f7e-4bf1-915d-1d0459b00edb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:50:20.957118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:20.957097    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2" pod "89c94084-6f7e-4bf1-915d-1d0459b00edb" (UID: "89c94084-6f7e-4bf1-915d-1d0459b00edb"). InnerVolumeSpecName "pvc-87143375-2cc3-4c8f-98e5-576e4b99838a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:50:21.055387 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.055307    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjdx8\" (UniqueName: \"kubernetes.io/projected/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-api-access-qjdx8\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:21.055387 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.055340    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c94084-6f7e-4bf1-915d-1d0459b00edb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:21.055387 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.055350    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c94084-6f7e-4bf1-915d-1d0459b00edb-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:21.818571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.818500    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"
Oct 31 07:50:21.818942 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.818488    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd" event={"ID":"89c94084-6f7e-4bf1-915d-1d0459b00edb","Type":"ContainerDied","Data":"9f645a3c799c435810c4d43b966deaf77373a01093a2b1ca025af87a2e8a0b2e"}
Oct 31 07:50:21.818942 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.818633    4913 scope.go:122] "RemoveContainer" containerID="0200bb7cc46c7f260c5d6ec2394849f3d32cfd42f9df662f9e716478e45b5411"
Oct 31 07:50:21.826925 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.826906    4913 scope.go:122] "RemoveContainer" containerID="eb2b45ac4ae24c1ecc03d3981512b1b240f3d4c08f3c7aece0a8b9bf0fb990af"
Oct 31 07:50:21.835229 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.835206    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"]
Oct 31 07:50:21.838710 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:21.838688    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-7c76bd9b9-pwjbd"]
Oct 31 07:50:23.036969 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:23.036931    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"]
Oct 31 07:50:23.037385 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:23.037158    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck" podUID="6b08558a-0c91-415d-82b9-2ea0d3d43a5e" containerName="kserve-container" containerID="cri-o://a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3" gracePeriod=30
Oct 31 07:50:23.124109 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:23.124075    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"]
Oct 31 07:50:23.124352 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:23.124309    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kserve-container" containerID="cri-o://97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f" gracePeriod=30
Oct 31 07:50:23.124531 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:23.124484    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy" containerID="cri-o://cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f" gracePeriod=30
Oct 31 07:50:23.419382 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:23.419305    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" path="/var/lib/kubelet/pods/89c94084-6f7e-4bf1-915d-1d0459b00edb/volumes"
Oct 31 07:50:23.830430 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:23.830398    4913 generic.go:356] "Generic (PLEG): container finished" podID="7e03291a-ede6-438c-a841-2248aa463f0c" containerID="cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f" exitCode=0
Oct 31 07:50:23.830614 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:23.830469    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" event={"ID":"7e03291a-ede6-438c-a841-2248aa463f0c","Type":"ContainerDied","Data":"cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f"}
Oct 31 07:50:25.667498 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:25.667458    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Oct 31 07:50:30.667438 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:30.667390    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Oct 31 07:50:35.666999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:35.666958    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Oct 31 07:50:35.667492 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:35.667079    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:50:40.667380 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:40.667337    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Oct 31 07:50:45.667115 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:45.667076    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Oct 31 07:50:49.399881 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:50:49.399845    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:50:50.666976 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:50.666928    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Oct 31 07:50:53.222599 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.222571    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:50:53.324945 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.324861    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b08558a-0c91-415d-82b9-2ea0d3d43a5e-kube-api-access-jc4k5\" (UniqueName: \"kubernetes.io/projected/6b08558a-0c91-415d-82b9-2ea0d3d43a5e-kube-api-access-jc4k5\") pod \"6b08558a-0c91-415d-82b9-2ea0d3d43a5e\" (UID: \"6b08558a-0c91-415d-82b9-2ea0d3d43a5e\") "
Oct 31 07:50:53.327755 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.327723    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b08558a-0c91-415d-82b9-2ea0d3d43a5e-kube-api-access-jc4k5" pod "6b08558a-0c91-415d-82b9-2ea0d3d43a5e" (UID: "6b08558a-0c91-415d-82b9-2ea0d3d43a5e"). InnerVolumeSpecName "kube-api-access-jc4k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:50:53.343371 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.343345    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:50:53.380281 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380250    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"]
Oct 31 07:50:53.380560 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380548    4913 cpu_manager.go:414] "containerMap: removing container" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kserve-container"
Oct 31 07:50:53.380603 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380561    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kserve-container"
Oct 31 07:50:53.380603 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380572    4913 cpu_manager.go:414] "containerMap: removing container" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy"
Oct 31 07:50:53.380603 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380578    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy"
Oct 31 07:50:53.380603 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380595    4913 cpu_manager.go:414] "containerMap: removing container" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kserve-container"
Oct 31 07:50:53.380603 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380601    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kserve-container"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380611    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6b08558a-0c91-415d-82b9-2ea0d3d43a5e" containerName="kserve-container"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380616    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b08558a-0c91-415d-82b9-2ea0d3d43a5e" containerName="kserve-container"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380628    4913 cpu_manager.go:414] "containerMap: removing container" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380633    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380685    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kube-rbac-proxy"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380695    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kserve-container"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380700    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" containerName="kube-rbac-proxy"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380709    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c94084-6f7e-4bf1-915d-1d0459b00edb" containerName="kserve-container"
Oct 31 07:50:53.380750 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.380715    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b08558a-0c91-415d-82b9-2ea0d3d43a5e" containerName="kserve-container"
Oct 31 07:50:53.383805 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.383783    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.387315 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.387290    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Oct 31 07:50:53.387646 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.387627    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:50:53.387739 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.387689    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:50:53.387739 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.387691    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:50:53.388098 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.388081    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-predictor-serving-cert\""
Oct 31 07:50:53.388253 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.388240    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:50:53.388314 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.388271    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:50:53.395672 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.395638    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"]
Oct 31 07:50:53.428975 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.428932    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") pod \"7e03291a-ede6-438c-a841-2248aa463f0c\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") "
Oct 31 07:50:53.429149 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429049    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e03291a-ede6-438c-a841-2248aa463f0c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e03291a-ede6-438c-a841-2248aa463f0c-proxy-tls\") pod \"7e03291a-ede6-438c-a841-2248aa463f0c\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") "
Oct 31 07:50:53.429149 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429084    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e03291a-ede6-438c-a841-2248aa463f0c-kube-api-access-r6hqs\" (UniqueName: \"kubernetes.io/projected/7e03291a-ede6-438c-a841-2248aa463f0c-kube-api-access-r6hqs\") pod \"7e03291a-ede6-438c-a841-2248aa463f0c\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") "
Oct 31 07:50:53.429149 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429139    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e03291a-ede6-438c-a841-2248aa463f0c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e03291a-ede6-438c-a841-2248aa463f0c-kube-rbac-proxy-sar-config\") pod \"7e03291a-ede6-438c-a841-2248aa463f0c\" (UID: \"7e03291a-ede6-438c-a841-2248aa463f0c\") "
Oct 31 07:50:53.429331 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429303    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkx5w\" (UniqueName: \"kubernetes.io/projected/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-api-access-kkx5w\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.429390 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429334    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9192c4-f216-4608-ba76-14b71e92d6dc-kserve-provision-location\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.429462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429433    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-cabundle-cert\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.429626 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429547    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9192c4-f216-4608-ba76-14b71e92d6dc-proxy-tls\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.429626 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429621    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.429753 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429709    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e03291a-ede6-438c-a841-2248aa463f0c-kube-rbac-proxy-sar-config" pod "7e03291a-ede6-438c-a841-2248aa463f0c" (UID: "7e03291a-ede6-438c-a841-2248aa463f0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:50:53.429817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.429763    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc4k5\" (UniqueName: \"kubernetes.io/projected/6b08558a-0c91-415d-82b9-2ea0d3d43a5e-kube-api-access-jc4k5\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:53.431840 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.431780    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e03291a-ede6-438c-a841-2248aa463f0c-proxy-tls" pod "7e03291a-ede6-438c-a841-2248aa463f0c" (UID: "7e03291a-ede6-438c-a841-2248aa463f0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:50:53.431929 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.431838    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e03291a-ede6-438c-a841-2248aa463f0c-kube-api-access-r6hqs" pod "7e03291a-ede6-438c-a841-2248aa463f0c" (UID: "7e03291a-ede6-438c-a841-2248aa463f0c"). InnerVolumeSpecName "kube-api-access-r6hqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:50:53.432941 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.432921    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2" pod "7e03291a-ede6-438c-a841-2248aa463f0c" (UID: "7e03291a-ede6-438c-a841-2248aa463f0c"). InnerVolumeSpecName "pvc-87143375-2cc3-4c8f-98e5-576e4b99838a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:50:53.530784 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.530749    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.530952 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.530804    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkx5w\" (UniqueName: \"kubernetes.io/projected/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-api-access-kkx5w\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.530952 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.530895    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9192c4-f216-4608-ba76-14b71e92d6dc-kserve-provision-location\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.531073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.530949    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-cabundle-cert\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.531073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.531002    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9192c4-f216-4608-ba76-14b71e92d6dc-proxy-tls\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.531138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.531070    4913 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" "
Oct 31 07:50:53.531138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.531088    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e03291a-ede6-438c-a841-2248aa463f0c-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:53.531138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.531102    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6hqs\" (UniqueName: \"kubernetes.io/projected/7e03291a-ede6-438c-a841-2248aa463f0c-kube-api-access-r6hqs\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:53.531138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.531115    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e03291a-ede6-438c-a841-2248aa463f0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:50:53.531357 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.531332    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9192c4-f216-4608-ba76-14b71e92d6dc-kserve-provision-location\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.531605 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.531581    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.531664 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.531589    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-cabundle-cert\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.533574 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.533554    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9192c4-f216-4608-ba76-14b71e92d6dc-proxy-tls\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.538399 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.538381    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkx5w\" (UniqueName: \"kubernetes.io/projected/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-api-access-kkx5w\") pod \"isvc-sklearn-predictor-5fb6797f48-qp24l\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.541693 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.541677    4913 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-87143375-2cc3-4c8f-98e5-576e4b99838a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2") on node "ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:50:53.632522 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.632442    4913 reconciler_common.go:299] "Volume detached for volume \"pvc-87143375-2cc3-4c8f-98e5-576e4b99838a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0deafdf927b335fb2\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"csi-6b020105ec0ae3d8191bba9389100371262bb677ec41338209cf16e393dfb259\""
Oct 31 07:50:53.704931 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.704902    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:53.955679 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.955636    4913 generic.go:356] "Generic (PLEG): container finished" podID="7e03291a-ede6-438c-a841-2248aa463f0c" containerID="97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f" exitCode=137
Oct 31 07:50:53.955856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.955694    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" event={"ID":"7e03291a-ede6-438c-a841-2248aa463f0c","Type":"ContainerDied","Data":"97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f"}
Oct 31 07:50:53.955856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.955711    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"
Oct 31 07:50:53.955856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.955771    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h" event={"ID":"7e03291a-ede6-438c-a841-2248aa463f0c","Type":"ContainerDied","Data":"1aec602339aa7cb568af62d44f359c47dd11f5f126e5598ad9e952228bfe55df"}
Oct 31 07:50:53.955856 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.955801    4913 scope.go:122] "RemoveContainer" containerID="cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f"
Oct 31 07:50:53.957487 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.957461    4913 generic.go:356] "Generic (PLEG): container finished" podID="6b08558a-0c91-415d-82b9-2ea0d3d43a5e" containerID="a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3" exitCode=137
Oct 31 07:50:53.957636 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.957526    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"
Oct 31 07:50:53.957636 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.957560    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck" event={"ID":"6b08558a-0c91-415d-82b9-2ea0d3d43a5e","Type":"ContainerDied","Data":"a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3"}
Oct 31 07:50:53.957636 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.957589    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck" event={"ID":"6b08558a-0c91-415d-82b9-2ea0d3d43a5e","Type":"ContainerDied","Data":"cbe1457f538e7659c36d85355c93293abeb1aa9a11e2a5821306f1b2816951ad"}
Oct 31 07:50:53.965158 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.965140    4913 scope.go:122] "RemoveContainer" containerID="97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f"
Oct 31 07:50:53.972813 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.972793    4913 scope.go:122] "RemoveContainer" containerID="cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f"
Oct 31 07:50:53.973113 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:50:53.973089    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f\": container with ID starting with cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f not found: ID does not exist" containerID="cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f"
Oct 31 07:50:53.973198 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.973126    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f"} err="failed to get container status \"cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f\": rpc error: code = NotFound desc = could not find container \"cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f\": container with ID starting with cad072c5af3ce7bc13698f960abde7225d00d09aaa05589b9abee3984817ec0f not found: ID does not exist"
Oct 31 07:50:53.973198 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.973150    4913 scope.go:122] "RemoveContainer" containerID="97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f"
Oct 31 07:50:53.973557 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:50:53.973535    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f\": container with ID starting with 97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f not found: ID does not exist" containerID="97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f"
Oct 31 07:50:53.973650 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.973565    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f"} err="failed to get container status \"97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f\": rpc error: code = NotFound desc = could not find container \"97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f\": container with ID starting with 97f9b43074c52cf2ae673a08ec7d51b5848a280a1882e267673cce503c63390f not found: ID does not exist"
Oct 31 07:50:53.973650 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.973612    4913 scope.go:122] "RemoveContainer" containerID="a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3"
Oct 31 07:50:53.974813 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.974793    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"]
Oct 31 07:50:53.977746 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.977726    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"]
Oct 31 07:50:53.981671 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.981647    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-sink-884fd44d9-brfck"]
Oct 31 07:50:53.982164 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.982148    4913 scope.go:122] "RemoveContainer" containerID="a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3"
Oct 31 07:50:53.982451 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:50:53.982432    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3\": container with ID starting with a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3 not found: ID does not exist" containerID="a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3"
Oct 31 07:50:53.982508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.982458    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3"} err="failed to get container status \"a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3\": rpc error: code = NotFound desc = could not find container \"a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3\": container with ID starting with a88fe5830b3cbc08eae863550572c2c28b0eb6f7b1f43500563bd6f1baad13f3 not found: ID does not exist"
Oct 31 07:50:53.983393 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:50:53.983371    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9192c4_f216_4608_ba76_14b71e92d6dc.slice/crio-d67982342d1e3ea18cc9f122a3a5c6b959359b03f89e2dd7a1b879f72a6c181f WatchSource:0}: Error finding container d67982342d1e3ea18cc9f122a3a5c6b959359b03f89e2dd7a1b879f72a6c181f: Status 404 returned error can't find the container with id d67982342d1e3ea18cc9f122a3a5c6b959359b03f89e2dd7a1b879f72a6c181f
Oct 31 07:50:53.992146 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.992124    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"]
Oct 31 07:50:53.994655 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:53.994612    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e5465674-predictor-6d54d9bccf-6qx8h"]
Oct 31 07:50:54.966217 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:54.966160    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" event={"ID":"2f9192c4-f216-4608-ba76-14b71e92d6dc","Type":"ContainerStarted","Data":"e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa"}
Oct 31 07:50:54.966618 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:54.966253    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" event={"ID":"2f9192c4-f216-4608-ba76-14b71e92d6dc","Type":"ContainerStarted","Data":"d67982342d1e3ea18cc9f122a3a5c6b959359b03f89e2dd7a1b879f72a6c181f"}
Oct 31 07:50:55.419706 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:55.419625    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b08558a-0c91-415d-82b9-2ea0d3d43a5e" path="/var/lib/kubelet/pods/6b08558a-0c91-415d-82b9-2ea0d3d43a5e/volumes"
Oct 31 07:50:55.420087 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:55.420072    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e03291a-ede6-438c-a841-2248aa463f0c" path="/var/lib/kubelet/pods/7e03291a-ede6-438c-a841-2248aa463f0c/volumes"
Oct 31 07:50:55.944528 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:55.944496    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"]
Oct 31 07:50:55.949131 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:55.949107    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:50:55.954403 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:55.954378    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:50:55.954525 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:55.954444    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:50:55.954525 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:55.954472    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:50:55.955847 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:55.955825    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"]
Oct 31 07:50:56.050932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.050894    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv59l\" (UniqueName: \"kubernetes.io/projected/4ab4d1fc-e013-4aa2-9573-ea7b473bb868-kube-api-access-fv59l\") pod \"proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9\" (UID: \"4ab4d1fc-e013-4aa2-9573-ea7b473bb868\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:50:56.152170 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.152136    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv59l\" (UniqueName: \"kubernetes.io/projected/4ab4d1fc-e013-4aa2-9573-ea7b473bb868-kube-api-access-fv59l\") pod \"proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9\" (UID: \"4ab4d1fc-e013-4aa2-9573-ea7b473bb868\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:50:56.159873 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.159830    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv59l\" (UniqueName: \"kubernetes.io/projected/4ab4d1fc-e013-4aa2-9573-ea7b473bb868-kube-api-access-fv59l\") pod \"proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9\" (UID: \"4ab4d1fc-e013-4aa2-9573-ea7b473bb868\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:50:56.264137 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.264046    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:50:56.531041 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.530954    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"]
Oct 31 07:50:56.533938 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:50:56.533910    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab4d1fc_e013_4aa2_9573_ea7b473bb868.slice/crio-330bdba0877d8787028fd4acc6889f34f55a27fc337984f65e2117282818bff9 WatchSource:0}: Error finding container 330bdba0877d8787028fd4acc6889f34f55a27fc337984f65e2117282818bff9: Status 404 returned error can't find the container with id 330bdba0877d8787028fd4acc6889f34f55a27fc337984f65e2117282818bff9
Oct 31 07:50:56.976072 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.976033    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9" event={"ID":"4ab4d1fc-e013-4aa2-9573-ea7b473bb868","Type":"ContainerStarted","Data":"e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d"}
Oct 31 07:50:56.976258 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.976093    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9" event={"ID":"4ab4d1fc-e013-4aa2-9573-ea7b473bb868","Type":"ContainerStarted","Data":"330bdba0877d8787028fd4acc6889f34f55a27fc337984f65e2117282818bff9"}
Oct 31 07:50:56.976258 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.976239    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:50:56.978667 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.978634    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:50:56.990984 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:56.990934    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9" podStartSLOduration=1.990922297 podStartE2EDuration="1.990922297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:50:56.990219425 +0000 UTC m=+1305.679987718" watchObservedRunningTime="2026-10-31 07:50:56.990922297 +0000 UTC m=+1305.680690590"
Oct 31 07:50:58.986325 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:58.986292    4913 generic.go:356] "Generic (PLEG): container finished" podID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerID="e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa" exitCode=0
Oct 31 07:50:58.986687 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:58.986365    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" event={"ID":"2f9192c4-f216-4608-ba76-14b71e92d6dc","Type":"ContainerDied","Data":"e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa"}
Oct 31 07:50:59.991958 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:59.991925    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" event={"ID":"2f9192c4-f216-4608-ba76-14b71e92d6dc","Type":"ContainerStarted","Data":"68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21"}
Oct 31 07:50:59.992402 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:59.991989    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" event={"ID":"2f9192c4-f216-4608-ba76-14b71e92d6dc","Type":"ContainerStarted","Data":"86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b"}
Oct 31 07:50:59.992402 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:59.992229    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:59.992402 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:59.992292    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:50:59.993817 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:50:59.993789    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:51:00.011040 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:00.010993    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podStartSLOduration=3.010637806 podStartE2EDuration="7.010979876s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.00034207s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:51:00.009931086 +0000 UTC m=+1308.699699383" watchObservedRunningTime="2026-10-31 07:51:00.010979876 +0000 UTC m=+1308.700748172"
Oct 31 07:51:00.996433 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:00.996392    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:51:01.284391 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.284309    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"]
Oct 31 07:51:01.287780 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.287761    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.291117 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.291092    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-e6a512bf-kube-rbac-proxy-sar-config\""
Oct 31 07:51:01.292266 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.292248    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-e6a512bf-predictor-serving-cert\""
Oct 31 07:51:01.292266 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.292259    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"proxy-routing-e6a512bf-dockercfg-2tc42\""
Oct 31 07:51:01.297299 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.297277    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"]
Oct 31 07:51:01.397872 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.397838    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6684edb-decf-4f09-a0c3-4b1e1d438b46-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.398040 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.397884    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxbx\" (UniqueName: \"kubernetes.io/projected/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-api-access-rjxbx\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.398040 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.398016    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.498746 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.498707    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6684edb-decf-4f09-a0c3-4b1e1d438b46-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.498918 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.498757    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxbx\" (UniqueName: \"kubernetes.io/projected/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-api-access-rjxbx\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.498918 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.498799    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.499475 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.499450    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.501293 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.501270    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6684edb-decf-4f09-a0c3-4b1e1d438b46-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:01.506347 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:01.506325    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxbx\" (UniqueName: \"kubernetes.io/projected/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-api-access-rjxbx\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:06.001180 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:06.001146    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:51:06.001768 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:06.001741    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:51:06.843127 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:06.843091    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:06.848810 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:06.848790    4913 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") device path: \"\"" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:06.944426 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:06.944388    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:06.944597 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:06.944529    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") DevicePath \"\"" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:06.947699 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:06.947680    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") DevicePath \"csi-d2c03b7a264f075f6e6a4ad12785d7e31f2cc22c92b94ec35e90ebdab38a0c46\"" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:07.114214 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:07.114108    4913 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1a6f6e56ad6de39df71a4f2d25a3c8eb0c0f738aa760b069aaefcb11ab8b6e1c/globalmount\"" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:07.126931 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:07.126904    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-6946c56ccd-v529h\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:07.305031 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:07.304989    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:07.579570 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:07.579504    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"]
Oct 31 07:51:08.027655 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:08.027621    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" event={"ID":"c6684edb-decf-4f09-a0c3-4b1e1d438b46","Type":"ContainerStarted","Data":"ad6107c47ac1bf2defc8fb51ee39ea7450fdaeacc0a2b68d40e54730210d6053"}
Oct 31 07:51:08.027830 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:08.027678    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" event={"ID":"c6684edb-decf-4f09-a0c3-4b1e1d438b46","Type":"ContainerStarted","Data":"1197b9534c750d76526e3f2c07159c8b0c6a82128642f74390b8e9edec7a6187"}
Oct 31 07:51:08.027830 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:08.027708    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" event={"ID":"c6684edb-decf-4f09-a0c3-4b1e1d438b46","Type":"ContainerStarted","Data":"bb15afbc321851db35fc3a8265a100b3d5dad0db51bf3ef3c42cd5d55dd11de8"}
Oct 31 07:51:08.027830 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:08.027734    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" event={"ID":"c6684edb-decf-4f09-a0c3-4b1e1d438b46","Type":"ContainerStarted","Data":"33205ff8dc457a95b2ec4813096a2c3b8296ac2e0c12095310ade800c4aa698a"}
Oct 31 07:51:08.027830 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:08.027793    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:08.045576 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:08.045526    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podStartSLOduration=10.045505165 podStartE2EDuration="10.045505165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:51:08.04451283 +0000 UTC m=+1316.734281133" watchObservedRunningTime="2026-10-31 07:51:08.045505165 +0000 UTC m=+1316.735273456"
Oct 31 07:51:09.032421 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:09.032391    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:09.032828 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:09.032453    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:09.034682 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:09.034656    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:09.035244 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:09.035225    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:16.002416 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:16.002325    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:51:16.045335 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:16.045306    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:18.312081 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.312050    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"]
Oct 31 07:51:18.316854 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.316833    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.326263 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.326239    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"]
Oct 31 07:51:18.337886 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.337860    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.338029 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.337897    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd23752-592a-4306-9b5b-80e8a059f625-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.338029 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.337923    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd23752-592a-4306-9b5b-80e8a059f625-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.338029 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.337999    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7ts\" (UniqueName: \"kubernetes.io/projected/2fd23752-592a-4306-9b5b-80e8a059f625-kube-api-access-pp7ts\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.338212 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.338028    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") DevicePath \"csi-d2c03b7a264f075f6e6a4ad12785d7e31f2cc22c92b94ec35e90ebdab38a0c46\"" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.340744 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.340722    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") DevicePath \"csi-d2c03b7a264f075f6e6a4ad12785d7e31f2cc22c92b94ec35e90ebdab38a0c46\"" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.353667 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.353642    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.439300 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.439251    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd23752-592a-4306-9b5b-80e8a059f625-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.439502 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.439389    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd23752-592a-4306-9b5b-80e8a059f625-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.439502 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.439444    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7ts\" (UniqueName: \"kubernetes.io/projected/2fd23752-592a-4306-9b5b-80e8a059f625-kube-api-access-pp7ts\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.439975 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.439956    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd23752-592a-4306-9b5b-80e8a059f625-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.441882 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.441858    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd23752-592a-4306-9b5b-80e8a059f625-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.446492 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.446467    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7ts\" (UniqueName: \"kubernetes.io/projected/2fd23752-592a-4306-9b5b-80e8a059f625-kube-api-access-pp7ts\") pod \"proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.635168 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.635080    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:18.907293 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:18.907220    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"]
Oct 31 07:51:18.910646 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:51:18.910622    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd23752_592a_4306_9b5b_80e8a059f625.slice/crio-43cbd4476246078b681744e8d1b99127a99b9f94cb53c36f9fe5754f986ab64b WatchSource:0}: Error finding container 43cbd4476246078b681744e8d1b99127a99b9f94cb53c36f9fe5754f986ab64b: Status 404 returned error can't find the container with id 43cbd4476246078b681744e8d1b99127a99b9f94cb53c36f9fe5754f986ab64b
Oct 31 07:51:19.083799 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:19.083772    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" event={"ID":"2fd23752-592a-4306-9b5b-80e8a059f625","Type":"ContainerStarted","Data":"0d7f254d5f5c5c81e3da895428b735546a3feafb08619a7583f48f5248c7ac61"}
Oct 31 07:51:19.083913 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:19.083851    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" event={"ID":"2fd23752-592a-4306-9b5b-80e8a059f625","Type":"ContainerStarted","Data":"1a1cb8e27772aa03d447fd47101390e0d5b8ba13a1213a477f123f558757b6da"}
Oct 31 07:51:19.083913 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:19.083875    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" event={"ID":"2fd23752-592a-4306-9b5b-80e8a059f625","Type":"ContainerStarted","Data":"43cbd4476246078b681744e8d1b99127a99b9f94cb53c36f9fe5754f986ab64b"}
Oct 31 07:51:19.399881 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:51:19.399843    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:51:20.091396 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:20.091359    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" event={"ID":"2fd23752-592a-4306-9b5b-80e8a059f625","Type":"ContainerStarted","Data":"e089064b1264cf4e3058320d9e4fdb28dca9ee7359ae621ab516f35535c7e448"}
Oct 31 07:51:20.091594 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:20.091562    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:20.091654 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:20.091610    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:20.091654 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:20.091642    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:20.094241 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:20.094218    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:20.094671 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:20.094647    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:20.110929 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:20.110882    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podStartSLOduration=2.110870643 podStartE2EDuration="2.110870643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:51:20.10909939 +0000 UTC m=+1328.798867682" watchObservedRunningTime="2026-10-31 07:51:20.110870643 +0000 UTC m=+1328.800639005"
Oct 31 07:51:26.001781 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:26.001737    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:51:27.105510 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:27.105482    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:51:27.154022 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:27.153968    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"]
Oct 31 07:51:27.154329 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:27.154298    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kserve-container" containerID="cri-o://bb15afbc321851db35fc3a8265a100b3d5dad0db51bf3ef3c42cd5d55dd11de8" gracePeriod=30
Oct 31 07:51:27.154726 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:27.154695    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="agent" containerID="cri-o://ad6107c47ac1bf2defc8fb51ee39ea7450fdaeacc0a2b68d40e54730210d6053" gracePeriod=30
Oct 31 07:51:27.154886 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:27.154818    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy" containerID="cri-o://1197b9534c750d76526e3f2c07159c8b0c6a82128642f74390b8e9edec7a6187" gracePeriod=30
Oct 31 07:51:28.128786 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:28.128751    4913 generic.go:356] "Generic (PLEG): container finished" podID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerID="1197b9534c750d76526e3f2c07159c8b0c6a82128642f74390b8e9edec7a6187" exitCode=0
Oct 31 07:51:28.129138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:28.128823    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" event={"ID":"c6684edb-decf-4f09-a0c3-4b1e1d438b46","Type":"ContainerDied","Data":"1197b9534c750d76526e3f2c07159c8b0c6a82128642f74390b8e9edec7a6187"}
Oct 31 07:51:29.033103 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:29.033060    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:51:31.041290 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:31.041249    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Oct 31 07:51:36.001788 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:36.001738    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:51:36.041280 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:36.041247    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Oct 31 07:51:39.033130 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:39.033083    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:51:41.041776 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:41.041736    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Oct 31 07:51:41.042223 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:41.041886    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:46.001725 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:46.001683    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:51:46.041564 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:46.041524    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Oct 31 07:51:49.033280 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:49.033227    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:51:49.033726 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:49.033365    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:49.400196 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:51:49.400103    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:51:51.041871 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:51.041830    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Oct 31 07:51:56.001924 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:56.001887    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:51:56.041905 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:56.041868    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Oct 31 07:51:57.259508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.259470    4913 generic.go:356] "Generic (PLEG): container finished" podID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerID="ad6107c47ac1bf2defc8fb51ee39ea7450fdaeacc0a2b68d40e54730210d6053" exitCode=0
Oct 31 07:51:57.259508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.259496    4913 generic.go:356] "Generic (PLEG): container finished" podID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerID="bb15afbc321851db35fc3a8265a100b3d5dad0db51bf3ef3c42cd5d55dd11de8" exitCode=137
Oct 31 07:51:57.259897 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.259540    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" event={"ID":"c6684edb-decf-4f09-a0c3-4b1e1d438b46","Type":"ContainerDied","Data":"ad6107c47ac1bf2defc8fb51ee39ea7450fdaeacc0a2b68d40e54730210d6053"}
Oct 31 07:51:57.259897 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.259621    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" event={"ID":"c6684edb-decf-4f09-a0c3-4b1e1d438b46","Type":"ContainerDied","Data":"bb15afbc321851db35fc3a8265a100b3d5dad0db51bf3ef3c42cd5d55dd11de8"}
Oct 31 07:51:57.814664 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.814638    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:57.876641 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.876608    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") "
Oct 31 07:51:57.876809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.876650    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6684edb-decf-4f09-a0c3-4b1e1d438b46-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6684edb-decf-4f09-a0c3-4b1e1d438b46-proxy-tls\") pod \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") "
Oct 31 07:51:57.876809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.876682    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-rbac-proxy-sar-config\") pod \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") "
Oct 31 07:51:57.876809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.876729    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-api-access-rjxbx\" (UniqueName: \"kubernetes.io/projected/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-api-access-rjxbx\") pod \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\" (UID: \"c6684edb-decf-4f09-a0c3-4b1e1d438b46\") "
Oct 31 07:51:57.877084 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.877047    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-rbac-proxy-sar-config" pod "c6684edb-decf-4f09-a0c3-4b1e1d438b46" (UID: "c6684edb-decf-4f09-a0c3-4b1e1d438b46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:51:57.878878 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.878850    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6684edb-decf-4f09-a0c3-4b1e1d438b46-proxy-tls" pod "c6684edb-decf-4f09-a0c3-4b1e1d438b46" (UID: "c6684edb-decf-4f09-a0c3-4b1e1d438b46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:51:57.878970 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.878957    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-api-access-rjxbx" pod "c6684edb-decf-4f09-a0c3-4b1e1d438b46" (UID: "c6684edb-decf-4f09-a0c3-4b1e1d438b46"). InnerVolumeSpecName "kube-api-access-rjxbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:51:57.879414 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.879391    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318" pod "c6684edb-decf-4f09-a0c3-4b1e1d438b46" (UID: "c6684edb-decf-4f09-a0c3-4b1e1d438b46"). InnerVolumeSpecName "pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:51:57.977907 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.977809    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6684edb-decf-4f09-a0c3-4b1e1d438b46-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:51:57.977907 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.977857    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:51:57.977907 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:57.977871    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjxbx\" (UniqueName: \"kubernetes.io/projected/c6684edb-decf-4f09-a0c3-4b1e1d438b46-kube-api-access-rjxbx\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:51:58.266025 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.265951    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"
Oct 31 07:51:58.266454 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.265951    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h" event={"ID":"c6684edb-decf-4f09-a0c3-4b1e1d438b46","Type":"ContainerDied","Data":"33205ff8dc457a95b2ec4813096a2c3b8296ac2e0c12095310ade800c4aa698a"}
Oct 31 07:51:58.266454 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.266100    4913 scope.go:122] "RemoveContainer" containerID="ad6107c47ac1bf2defc8fb51ee39ea7450fdaeacc0a2b68d40e54730210d6053"
Oct 31 07:51:58.275379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.275361    4913 scope.go:122] "RemoveContainer" containerID="1197b9534c750d76526e3f2c07159c8b0c6a82128642f74390b8e9edec7a6187"
Oct 31 07:51:58.282838 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.282814    4913 scope.go:122] "RemoveContainer" containerID="bb15afbc321851db35fc3a8265a100b3d5dad0db51bf3ef3c42cd5d55dd11de8"
Oct 31 07:51:58.287613 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.287590    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"]
Oct 31 07:51:58.291411 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.291392    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-6946c56ccd-v529h"]
Oct 31 07:51:58.974701 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.974666    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"]
Oct 31 07:51:58.975022 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975010    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="agent"
Oct 31 07:51:58.975073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975023    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="agent"
Oct 31 07:51:58.975073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975050    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kserve-container"
Oct 31 07:51:58.975073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975056    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kserve-container"
Oct 31 07:51:58.975073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975063    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy"
Oct 31 07:51:58.975073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975068    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy"
Oct 31 07:51:58.975306 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975131    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kserve-container"
Oct 31 07:51:58.975306 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975138    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="kube-rbac-proxy"
Oct 31 07:51:58.975306 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.975148    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" containerName="agent"
Oct 31 07:51:58.980030 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.980009    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:58.988939 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:58.988912    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"]
Oct 31 07:51:59.088790 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.088755    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.088985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.088797    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jq5\" (UniqueName: \"kubernetes.io/projected/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-api-access-p2jq5\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.088985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.088956    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.089095 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.089007    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.089095 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.089039    4913 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") DevicePath \"csi-d2c03b7a264f075f6e6a4ad12785d7e31f2cc22c92b94ec35e90ebdab38a0c46\"" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.091523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.091500    4913 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") DevicePath \"csi-d2c03b7a264f075f6e6a4ad12785d7e31f2cc22c92b94ec35e90ebdab38a0c46\"" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.102177 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.102152    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.190476 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.190440    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.190672 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.190521    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.190672 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.190554    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jq5\" (UniqueName: \"kubernetes.io/projected/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-api-access-p2jq5\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.191262 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.191237    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-rbac-proxy-sar-config\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.192863 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.192845    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-proxy-tls\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.197763 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.197741    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jq5\" (UniqueName: \"kubernetes.io/projected/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-api-access-p2jq5\") pod \"proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") " pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.297358 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.297272    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:51:59.419906 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.419872    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6684edb-decf-4f09-a0c3-4b1e1d438b46" path="/var/lib/kubelet/pods/c6684edb-decf-4f09-a0c3-4b1e1d438b46/volumes"
Oct 31 07:51:59.568837 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:51:59.568814    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"]
Oct 31 07:51:59.571415 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:51:59.571388    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee2d8a8_ab36_4f8d_9c27_d21abfad53d4.slice/crio-8fa8757da8bcda6818918063209eb1e01b73cb666cbb305f790f4615ff57356a WatchSource:0}: Error finding container 8fa8757da8bcda6818918063209eb1e01b73cb666cbb305f790f4615ff57356a: Status 404 returned error can't find the container with id 8fa8757da8bcda6818918063209eb1e01b73cb666cbb305f790f4615ff57356a
Oct 31 07:52:00.276748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:00.276716    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" event={"ID":"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4","Type":"ContainerStarted","Data":"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419"}
Oct 31 07:52:00.276929 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:00.276776    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" event={"ID":"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4","Type":"ContainerStarted","Data":"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638"}
Oct 31 07:52:00.276929 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:00.276808    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" event={"ID":"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4","Type":"ContainerStarted","Data":"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084"}
Oct 31 07:52:00.276929 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:00.276832    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" event={"ID":"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4","Type":"ContainerStarted","Data":"8fa8757da8bcda6818918063209eb1e01b73cb666cbb305f790f4615ff57356a"}
Oct 31 07:52:00.276929 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:00.276878    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:52:00.296335 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:00.296277    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podStartSLOduration=2.296263559 podStartE2EDuration="2.296263559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:52:00.294493256 +0000 UTC m=+1368.984261549" watchObservedRunningTime="2026-10-31 07:52:00.296263559 +0000 UTC m=+1368.986031851"
Oct 31 07:52:01.281822 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:01.281790    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:52:01.282234 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:01.281891    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:52:01.284506 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:01.284481    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:52:01.284938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:01.284922    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:52:06.002352 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:06.002307    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:52:08.295548 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:08.295520    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:52:08.373815 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:08.373772    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"]
Oct 31 07:52:08.374082 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:08.374054    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kserve-container" containerID="cri-o://1a1cb8e27772aa03d447fd47101390e0d5b8ba13a1213a477f123f558757b6da" gracePeriod=30
Oct 31 07:52:08.374266 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:08.374238    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy" containerID="cri-o://0d7f254d5f5c5c81e3da895428b735546a3feafb08619a7583f48f5248c7ac61" gracePeriod=30
Oct 31 07:52:08.374883 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:08.374269    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="agent" containerID="cri-o://e089064b1264cf4e3058320d9e4fdb28dca9ee7359ae621ab516f35535c7e448" gracePeriod=30
Oct 31 07:52:09.316740 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:09.316704    4913 generic.go:356] "Generic (PLEG): container finished" podID="2fd23752-592a-4306-9b5b-80e8a059f625" containerID="0d7f254d5f5c5c81e3da895428b735546a3feafb08619a7583f48f5248c7ac61" exitCode=0
Oct 31 07:52:09.317124 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:09.316777    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" event={"ID":"2fd23752-592a-4306-9b5b-80e8a059f625","Type":"ContainerDied","Data":"0d7f254d5f5c5c81e3da895428b735546a3feafb08619a7583f48f5248c7ac61"}
Oct 31 07:52:10.092692 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:10.092644    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:52:12.101965 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:12.101918    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 31 07:52:16.001762 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:16.001724    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:52:17.101801 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:17.101751    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 31 07:52:17.419374 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:17.419303    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:52:19.400816 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:52:19.400778    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:52:20.092327 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:20.092287    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:52:22.101474 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:22.101437    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 31 07:52:22.101862 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:22.101571    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:52:24.053583 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:24.053530    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:26.077116 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:26.077085    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:27.102075 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:27.102039    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 31 07:52:28.100376 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:28.100347    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:30.092662 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:30.092611    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:52:30.093084 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:30.092774    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:52:30.123242 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:30.123212    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:32.101659 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:32.101620    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 31 07:52:32.146220 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:32.146174    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:34.170428 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:34.170396    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:36.193913 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:36.193882    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:37.101110 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:37.101063    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Oct 31 07:52:38.217501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.217473    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:38.443270 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.443221    4913 generic.go:356] "Generic (PLEG): container finished" podID="2fd23752-592a-4306-9b5b-80e8a059f625" containerID="e089064b1264cf4e3058320d9e4fdb28dca9ee7359ae621ab516f35535c7e448" exitCode=137
Oct 31 07:52:38.443270 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.443258    4913 generic.go:356] "Generic (PLEG): container finished" podID="2fd23752-592a-4306-9b5b-80e8a059f625" containerID="1a1cb8e27772aa03d447fd47101390e0d5b8ba13a1213a477f123f558757b6da" exitCode=137
Oct 31 07:52:38.443488 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.443277    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" event={"ID":"2fd23752-592a-4306-9b5b-80e8a059f625","Type":"ContainerDied","Data":"e089064b1264cf4e3058320d9e4fdb28dca9ee7359ae621ab516f35535c7e448"}
Oct 31 07:52:38.443488 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.443360    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" event={"ID":"2fd23752-592a-4306-9b5b-80e8a059f625","Type":"ContainerDied","Data":"1a1cb8e27772aa03d447fd47101390e0d5b8ba13a1213a477f123f558757b6da"}
Oct 31 07:52:38.445552 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:52:38.445523    4913 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd23752_592a_4306_9b5b_80e8a059f625.slice/crio-43cbd4476246078b681744e8d1b99127a99b9f94cb53c36f9fe5754f986ab64b\": RecentStats: unable to find data in memory cache]"
Oct 31 07:52:38.591966 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.591935    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:52:38.755961 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.755869    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd23752-592a-4306-9b5b-80e8a059f625-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd23752-592a-4306-9b5b-80e8a059f625-kube-rbac-proxy-sar-config\") pod \"2fd23752-592a-4306-9b5b-80e8a059f625\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") "
Oct 31 07:52:38.755961 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.755940    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd23752-592a-4306-9b5b-80e8a059f625-kube-api-access-pp7ts\" (UniqueName: \"kubernetes.io/projected/2fd23752-592a-4306-9b5b-80e8a059f625-kube-api-access-pp7ts\") pod \"2fd23752-592a-4306-9b5b-80e8a059f625\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") "
Oct 31 07:52:38.756236 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.756097    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"2fd23752-592a-4306-9b5b-80e8a059f625\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") "
Oct 31 07:52:38.756236 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.756155    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd23752-592a-4306-9b5b-80e8a059f625-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd23752-592a-4306-9b5b-80e8a059f625-proxy-tls\") pod \"2fd23752-592a-4306-9b5b-80e8a059f625\" (UID: \"2fd23752-592a-4306-9b5b-80e8a059f625\") "
Oct 31 07:52:38.756343 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.756312    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd23752-592a-4306-9b5b-80e8a059f625-kube-rbac-proxy-sar-config" pod "2fd23752-592a-4306-9b5b-80e8a059f625" (UID: "2fd23752-592a-4306-9b5b-80e8a059f625"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:52:38.756442 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.756426    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd23752-592a-4306-9b5b-80e8a059f625-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:52:38.758394 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.758356    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd23752-592a-4306-9b5b-80e8a059f625-kube-api-access-pp7ts" pod "2fd23752-592a-4306-9b5b-80e8a059f625" (UID: "2fd23752-592a-4306-9b5b-80e8a059f625"). InnerVolumeSpecName "kube-api-access-pp7ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:52:38.758508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.758425    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd23752-592a-4306-9b5b-80e8a059f625-proxy-tls" pod "2fd23752-592a-4306-9b5b-80e8a059f625" (UID: "2fd23752-592a-4306-9b5b-80e8a059f625"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:52:38.758894 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.758873    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318" pod "2fd23752-592a-4306-9b5b-80e8a059f625" (UID: "2fd23752-592a-4306-9b5b-80e8a059f625"). InnerVolumeSpecName "pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:52:38.857176 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.857135    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd23752-592a-4306-9b5b-80e8a059f625-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:52:38.857176 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:38.857171    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7ts\" (UniqueName: \"kubernetes.io/projected/2fd23752-592a-4306-9b5b-80e8a059f625-kube-api-access-pp7ts\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:52:39.450357 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:39.450323    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks" event={"ID":"2fd23752-592a-4306-9b5b-80e8a059f625","Type":"ContainerDied","Data":"43cbd4476246078b681744e8d1b99127a99b9f94cb53c36f9fe5754f986ab64b"}
Oct 31 07:52:39.450811 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:39.450391    4913 scope.go:122] "RemoveContainer" containerID="e089064b1264cf4e3058320d9e4fdb28dca9ee7359ae621ab516f35535c7e448"
Oct 31 07:52:39.450811 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:39.450336    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"
Oct 31 07:52:39.458728 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:39.458681    4913 scope.go:122] "RemoveContainer" containerID="0d7f254d5f5c5c81e3da895428b735546a3feafb08619a7583f48f5248c7ac61"
Oct 31 07:52:39.467357 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:39.467322    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"]
Oct 31 07:52:39.471617 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:39.471589    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-c574c7bd4-c6nks"]
Oct 31 07:52:39.472953 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:39.472937    4913 scope.go:122] "RemoveContainer" containerID="1a1cb8e27772aa03d447fd47101390e0d5b8ba13a1213a477f123f558757b6da"
Oct 31 07:52:40.240853 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:40.240826    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:40.797517 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:40.797487    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"]
Oct 31 07:52:40.797920 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:40.797700    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9" podUID="4ab4d1fc-e013-4aa2-9573-ea7b473bb868" containerName="kserve-container" containerID="cri-o://e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d" gracePeriod=30
Oct 31 07:52:40.903080 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:40.903047    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"]
Oct 31 07:52:40.903400 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:40.903371    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kserve-container" containerID="cri-o://e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084" gracePeriod=30
Oct 31 07:52:40.903473 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:40.903414    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy" containerID="cri-o://b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638" gracePeriod=30
Oct 31 07:52:40.903549 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:40.903406    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="agent" containerID="cri-o://c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419" gracePeriod=30
Oct 31 07:52:41.282230 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:41.282159    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:52:41.419335 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:41.419301    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" path="/var/lib/kubelet/pods/2fd23752-592a-4306-9b5b-80e8a059f625/volumes"
Oct 31 07:52:41.462230 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:41.462197    4913 generic.go:356] "Generic (PLEG): container finished" podID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerID="b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638" exitCode=0
Oct 31 07:52:41.462386 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:41.462258    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" event={"ID":"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4","Type":"ContainerDied","Data":"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638"}
Oct 31 07:52:42.264058 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:42.264026    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:43.291060 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:43.291015    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 31 07:52:44.287596 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:44.287564    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:46.311072 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:46.311038    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:48.291872 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:48.291826    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 31 07:52:48.334443 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:48.334415    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:49.400313 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:52:49.400271    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:52:50.357454 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:50.357426    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:51.282717 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:51.282675    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:52:52.380425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:52.380396    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:53.291302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:53.291265    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 31 07:52:53.291475 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:53.291402    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:52:54.411244 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:54.411213    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-predictor-5fb6797f48-qp24l_2f9192c4-f216-4608-ba76-14b71e92d6dc/kserve-container/0.log"
Oct 31 07:52:54.744843 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:54.744808    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"]
Oct 31 07:52:54.745088 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:54.745055    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" containerID="cri-o://86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b" gracePeriod=30
Oct 31 07:52:54.745229 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:54.745101    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kube-rbac-proxy" containerID="cri-o://68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21" gracePeriod=30
Oct 31 07:52:55.523756 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:55.523725    4913 generic.go:356] "Generic (PLEG): container finished" podID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerID="68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21" exitCode=0
Oct 31 07:52:55.524240 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:55.523797    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" event={"ID":"2f9192c4-f216-4608-ba76-14b71e92d6dc","Type":"ContainerDied","Data":"68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21"}
Oct 31 07:52:55.997480 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:55.997442    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Oct 31 07:52:57.414909 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:57.414860    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Oct 31 07:52:58.291901 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:58.291859    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 31 07:52:59.689978 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.689953    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:52:59.714229 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.714195    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9192c4-f216-4608-ba76-14b71e92d6dc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9192c4-f216-4608-ba76-14b71e92d6dc-kserve-provision-location\") pod \"2f9192c4-f216-4608-ba76-14b71e92d6dc\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") "
Oct 31 07:52:59.714394 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.714296    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-rbac-proxy-sar-config\") pod \"2f9192c4-f216-4608-ba76-14b71e92d6dc\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") "
Oct 31 07:52:59.714394 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.714329    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-cabundle-cert\") pod \"2f9192c4-f216-4608-ba76-14b71e92d6dc\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") "
Oct 31 07:52:59.714394 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.714368    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f9192c4-f216-4608-ba76-14b71e92d6dc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9192c4-f216-4608-ba76-14b71e92d6dc-proxy-tls\") pod \"2f9192c4-f216-4608-ba76-14b71e92d6dc\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") "
Oct 31 07:52:59.714565 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.714406    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-api-access-kkx5w\" (UniqueName: \"kubernetes.io/projected/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-api-access-kkx5w\") pod \"2f9192c4-f216-4608-ba76-14b71e92d6dc\" (UID: \"2f9192c4-f216-4608-ba76-14b71e92d6dc\") "
Oct 31 07:52:59.714565 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.714474    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9192c4-f216-4608-ba76-14b71e92d6dc-kserve-provision-location" pod "2f9192c4-f216-4608-ba76-14b71e92d6dc" (UID: "2f9192c4-f216-4608-ba76-14b71e92d6dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:52:59.714682 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.714652    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9192c4-f216-4608-ba76-14b71e92d6dc-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:52:59.714737 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.714721    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-cabundle-cert" pod "2f9192c4-f216-4608-ba76-14b71e92d6dc" (UID: "2f9192c4-f216-4608-ba76-14b71e92d6dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:52:59.715109 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.715083    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-rbac-proxy-sar-config" pod "2f9192c4-f216-4608-ba76-14b71e92d6dc" (UID: "2f9192c4-f216-4608-ba76-14b71e92d6dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:52:59.716651 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.716626    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9192c4-f216-4608-ba76-14b71e92d6dc-proxy-tls" pod "2f9192c4-f216-4608-ba76-14b71e92d6dc" (UID: "2f9192c4-f216-4608-ba76-14b71e92d6dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:52:59.716729 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.716629    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-api-access-kkx5w" pod "2f9192c4-f216-4608-ba76-14b71e92d6dc" (UID: "2f9192c4-f216-4608-ba76-14b71e92d6dc"). InnerVolumeSpecName "kube-api-access-kkx5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:52:59.815627 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.815544    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:52:59.815627 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.815577    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9192c4-f216-4608-ba76-14b71e92d6dc-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:52:59.815627 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.815588    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9192c4-f216-4608-ba76-14b71e92d6dc-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:52:59.815627 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:52:59.815598    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkx5w\" (UniqueName: \"kubernetes.io/projected/2f9192c4-f216-4608-ba76-14b71e92d6dc-kube-api-access-kkx5w\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:53:00.546396 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.546361    4913 generic.go:356] "Generic (PLEG): container finished" podID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerID="86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b" exitCode=0
Oct 31 07:53:00.546575 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.546424    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"
Oct 31 07:53:00.546575 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.546429    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" event={"ID":"2f9192c4-f216-4608-ba76-14b71e92d6dc","Type":"ContainerDied","Data":"86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b"}
Oct 31 07:53:00.546575 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.546545    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l" event={"ID":"2f9192c4-f216-4608-ba76-14b71e92d6dc","Type":"ContainerDied","Data":"d67982342d1e3ea18cc9f122a3a5c6b959359b03f89e2dd7a1b879f72a6c181f"}
Oct 31 07:53:00.546575 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.546570    4913 scope.go:122] "RemoveContainer" containerID="68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21"
Oct 31 07:53:00.555920 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.555903    4913 scope.go:122] "RemoveContainer" containerID="86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b"
Oct 31 07:53:00.565785 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.565632    4913 scope.go:122] "RemoveContainer" containerID="e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa"
Oct 31 07:53:00.568778 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.568757    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"]
Oct 31 07:53:00.572442 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.572419    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-predictor-5fb6797f48-qp24l"]
Oct 31 07:53:00.573938 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.573921    4913 scope.go:122] "RemoveContainer" containerID="68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21"
Oct 31 07:53:00.574256 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:00.574237    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21\": container with ID starting with 68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21 not found: ID does not exist" containerID="68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21"
Oct 31 07:53:00.574317 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.574264    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21"} err="failed to get container status \"68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21\": rpc error: code = NotFound desc = could not find container \"68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21\": container with ID starting with 68df73421b4387eb492abc81cf3ea89e81d4745bc4445bf5859bea5fb9fa9a21 not found: ID does not exist"
Oct 31 07:53:00.574317 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.574282    4913 scope.go:122] "RemoveContainer" containerID="86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b"
Oct 31 07:53:00.574512 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:00.574499    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b\": container with ID starting with 86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b not found: ID does not exist" containerID="86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b"
Oct 31 07:53:00.574558 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.574516    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b"} err="failed to get container status \"86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b\": rpc error: code = NotFound desc = could not find container \"86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b\": container with ID starting with 86592d8a1da75d9f4a046e03ddf2c414fd5f957c9b8c63c9897d7229288d8a4b not found: ID does not exist"
Oct 31 07:53:00.574558 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.574528    4913 scope.go:122] "RemoveContainer" containerID="e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa"
Oct 31 07:53:00.574745 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:00.574719    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa\": container with ID starting with e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa not found: ID does not exist" containerID="e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa"
Oct 31 07:53:00.574850 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:00.574752    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa"} err="failed to get container status \"e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa\": rpc error: code = NotFound desc = could not find container \"e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa\": container with ID starting with e26f5d09b96eafac37082f831ee27911237deed36bc95f535f3f64b52854ccaa not found: ID does not exist"
Oct 31 07:53:01.282906 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:01.282868    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:53:01.283376 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:01.283010    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:53:01.419850 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:01.419810    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" path="/var/lib/kubelet/pods/2f9192c4-f216-4608-ba76-14b71e92d6dc/volumes"
Oct 31 07:53:03.291334 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:03.291290    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 31 07:53:08.291494 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:08.291448    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Oct 31 07:53:10.990723 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:10.990701    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:53:11.106476 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.106391    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab4d1fc-e013-4aa2-9573-ea7b473bb868-kube-api-access-fv59l\" (UniqueName: \"kubernetes.io/projected/4ab4d1fc-e013-4aa2-9573-ea7b473bb868-kube-api-access-fv59l\") pod \"4ab4d1fc-e013-4aa2-9573-ea7b473bb868\" (UID: \"4ab4d1fc-e013-4aa2-9573-ea7b473bb868\") "
Oct 31 07:53:11.108425 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.108402    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab4d1fc-e013-4aa2-9573-ea7b473bb868-kube-api-access-fv59l" pod "4ab4d1fc-e013-4aa2-9573-ea7b473bb868" (UID: "4ab4d1fc-e013-4aa2-9573-ea7b473bb868"). InnerVolumeSpecName "kube-api-access-fv59l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:53:11.207739 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.207696    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv59l\" (UniqueName: \"kubernetes.io/projected/4ab4d1fc-e013-4aa2-9573-ea7b473bb868-kube-api-access-fv59l\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:53:11.282165 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.282124    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.55:8081/\": dial tcp 10.132.0.55:8081: connect: connection refused"
Oct 31 07:53:11.282357 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.282124    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="agent" probeResult="failure" output="Get \"http://10.132.0.55:9081/\": dial tcp 10.132.0.55:9081: connect: connection refused"
Oct 31 07:53:11.546792 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.546767    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:53:11.593808 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.593778    4913 generic.go:356] "Generic (PLEG): container finished" podID="4ab4d1fc-e013-4aa2-9573-ea7b473bb868" containerID="e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d" exitCode=137
Oct 31 07:53:11.593985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.593820    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"
Oct 31 07:53:11.593985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.593866    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9" event={"ID":"4ab4d1fc-e013-4aa2-9573-ea7b473bb868","Type":"ContainerDied","Data":"e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d"}
Oct 31 07:53:11.593985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.593915    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9" event={"ID":"4ab4d1fc-e013-4aa2-9573-ea7b473bb868","Type":"ContainerDied","Data":"330bdba0877d8787028fd4acc6889f34f55a27fc337984f65e2117282818bff9"}
Oct 31 07:53:11.593985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.593942    4913 scope.go:122] "RemoveContainer" containerID="e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d"
Oct 31 07:53:11.596041 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.595967    4913 generic.go:356] "Generic (PLEG): container finished" podID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerID="c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419" exitCode=137
Oct 31 07:53:11.596041 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.595992    4913 generic.go:356] "Generic (PLEG): container finished" podID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerID="e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084" exitCode=137
Oct 31 07:53:11.596041 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.596022    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" event={"ID":"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4","Type":"ContainerDied","Data":"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419"}
Oct 31 07:53:11.596407 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.596074    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"
Oct 31 07:53:11.596407 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.596078    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" event={"ID":"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4","Type":"ContainerDied","Data":"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084"}
Oct 31 07:53:11.596407 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.596370    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m" event={"ID":"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4","Type":"ContainerDied","Data":"8fa8757da8bcda6818918063209eb1e01b73cb666cbb305f790f4615ff57356a"}
Oct 31 07:53:11.602420 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.602390    4913 scope.go:122] "RemoveContainer" containerID="e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d"
Oct 31 07:53:11.602687 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:11.602669    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d\": container with ID starting with e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d not found: ID does not exist" containerID="e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d"
Oct 31 07:53:11.602745 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.602696    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d"} err="failed to get container status \"e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d\": rpc error: code = NotFound desc = could not find container \"e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d\": container with ID starting with e347ba41370c498bfa804db7dd834ba2ee74b2c8f57ad85fe948b03daa569f6d not found: ID does not exist"
Oct 31 07:53:11.602745 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.602715    4913 scope.go:122] "RemoveContainer" containerID="c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419"
Oct 31 07:53:11.608635 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.608616    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"]
Oct 31 07:53:11.610349 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.610324    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-rbac-proxy-sar-config\") pod \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") "
Oct 31 07:53:11.610455 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.610378    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-proxy-tls\") pod \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") "
Oct 31 07:53:11.610523 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.610462    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-api-access-p2jq5\" (UniqueName: \"kubernetes.io/projected/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-api-access-p2jq5\") pod \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") "
Oct 31 07:53:11.610657 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.610559    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") pod \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\" (UID: \"3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4\") "
Oct 31 07:53:11.610747 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.610723    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-rbac-proxy-sar-config" pod "3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" (UID: "3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:53:11.610828 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.610816    4913 scope.go:122] "RemoveContainer" containerID="b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638"
Oct 31 07:53:11.613443 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.612948    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-proxy-tls" pod "3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" (UID: "3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:53:11.619939 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.615447    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-sink-58b76bc4f5-qckr9"]
Oct 31 07:53:11.620381 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.620355    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-api-access-p2jq5" pod "3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" (UID: "3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4"). InnerVolumeSpecName "kube-api-access-p2jq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:53:11.620751 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.620730    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318" pod "3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" (UID: "3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4"). InnerVolumeSpecName "pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 07:53:11.623284 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.623262    4913 scope.go:122] "RemoveContainer" containerID="e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084"
Oct 31 07:53:11.633154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.633133    4913 scope.go:122] "RemoveContainer" containerID="c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419"
Oct 31 07:53:11.633429 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:11.633407    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419\": container with ID starting with c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419 not found: ID does not exist" containerID="c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419"
Oct 31 07:53:11.633490 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.633436    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419"} err="failed to get container status \"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419\": rpc error: code = NotFound desc = could not find container \"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419\": container with ID starting with c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419 not found: ID does not exist"
Oct 31 07:53:11.633490 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.633457    4913 scope.go:122] "RemoveContainer" containerID="b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638"
Oct 31 07:53:11.633705 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:11.633679    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638\": container with ID starting with b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638 not found: ID does not exist" containerID="b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638"
Oct 31 07:53:11.633787 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.633709    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638"} err="failed to get container status \"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638\": rpc error: code = NotFound desc = could not find container \"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638\": container with ID starting with b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638 not found: ID does not exist"
Oct 31 07:53:11.633787 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.633730    4913 scope.go:122] "RemoveContainer" containerID="e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084"
Oct 31 07:53:11.633959 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:11.633945    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084\": container with ID starting with e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084 not found: ID does not exist" containerID="e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084"
Oct 31 07:53:11.634004 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.633965    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084"} err="failed to get container status \"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084\": rpc error: code = NotFound desc = could not find container \"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084\": container with ID starting with e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084 not found: ID does not exist"
Oct 31 07:53:11.634004 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.633979    4913 scope.go:122] "RemoveContainer" containerID="c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419"
Oct 31 07:53:11.634240 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.634216    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419"} err="failed to get container status \"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419\": rpc error: code = NotFound desc = could not find container \"c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419\": container with ID starting with c8819425942cde51b07f7d435c9b3d0b80d22d99be17e6166e50a91f6e84c419 not found: ID does not exist"
Oct 31 07:53:11.634240 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.634240    4913 scope.go:122] "RemoveContainer" containerID="b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638"
Oct 31 07:53:11.634478 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.634461    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638"} err="failed to get container status \"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638\": rpc error: code = NotFound desc = could not find container \"b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638\": container with ID starting with b7b74ea1c1b39bf3363248faf0763afa7d7836b6e9a138b4c4b1f1f761f33638 not found: ID does not exist"
Oct 31 07:53:11.634478 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.634478    4913 scope.go:122] "RemoveContainer" containerID="e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084"
Oct 31 07:53:11.634768 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.634742    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084"} err="failed to get container status \"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084\": rpc error: code = NotFound desc = could not find container \"e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084\": container with ID starting with e10108eb91ca36185578c7afa616be7a58eb305b747d5ac75a5e573e0ae57084 not found: ID does not exist"
Oct 31 07:53:11.711877 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.711794    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jq5\" (UniqueName: \"kubernetes.io/projected/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-api-access-p2jq5\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:53:11.711877 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.711847    4913 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" "
Oct 31 07:53:11.711877 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.711864    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:53:11.711877 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.711875    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:53:11.722615 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.722587    4913 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318") on node "ip-10-0-130-121.us-east-2.compute.internal"
Oct 31 07:53:11.813145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.813019    4913 reconciler_common.go:299] "Volume detached for volume \"pvc-6e63f3f2-7c7f-487e-9806-a364b80470b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d3637cf9ab7ae318\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"csi-d2c03b7a264f075f6e6a4ad12785d7e31f2cc22c92b94ec35e90ebdab38a0c46\""
Oct 31 07:53:11.917373 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.917336    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"]
Oct 31 07:53:11.920986 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:11.920960    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/proxy-routing-e6a512bf-predictor-56b5cc4555-ft25m"]
Oct 31 07:53:13.419348 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:13.419311    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" path="/var/lib/kubelet/pods/3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4/volumes"
Oct 31 07:53:13.419901 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:13.419879    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab4d1fc-e013-4aa2-9573-ea7b473bb868" path="/var/lib/kubelet/pods/4ab4d1fc-e013-4aa2-9573-ea7b473bb868/volumes"
Oct 31 07:53:14.077157 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077123    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"]
Oct 31 07:53:14.077462 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077449    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy"
Oct 31 07:53:14.077508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077464    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy"
Oct 31 07:53:14.077508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077474    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="agent"
Oct 31 07:53:14.077508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077479    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="agent"
Oct 31 07:53:14.077508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077490    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="storage-initializer"
Oct 31 07:53:14.077508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077496    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="storage-initializer"
Oct 31 07:53:14.077508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077503    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kube-rbac-proxy"
Oct 31 07:53:14.077508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077508    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kube-rbac-proxy"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077515    4913 cpu_manager.go:414] "containerMap: removing container" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077526    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077539    4913 cpu_manager.go:414] "containerMap: removing container" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077544    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077559    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077564    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077570    4913 cpu_manager.go:414] "containerMap: removing container" podUID="4ab4d1fc-e013-4aa2-9573-ea7b473bb868" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077575    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab4d1fc-e013-4aa2-9573-ea7b473bb868" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077581    4913 cpu_manager.go:414] "containerMap: removing container" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="agent"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077585    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="agent"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077594    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077599    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077662    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077669    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab4d1fc-e013-4aa2-9573-ea7b473bb868" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077674    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="agent"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077681    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077688    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="kube-rbac-proxy"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077693    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd23752-592a-4306-9b5b-80e8a059f625" containerName="agent"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077698    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9192c4-f216-4608-ba76-14b71e92d6dc" containerName="kube-rbac-proxy"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077703    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kserve-container"
Oct 31 07:53:14.077713 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.077710    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee2d8a8-ab36-4f8d-9c27-d21abfad53d4" containerName="kube-rbac-proxy"
Oct 31 07:53:14.082486 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.082465    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.089090 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.088876    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Oct 31 07:53:14.089090 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.088910    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:53:14.089090 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.088881    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"message-dumper-predictor-serving-cert\""
Oct 31 07:53:14.089090 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.088991    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:53:14.089876 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.089536    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:53:14.090373 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.090343    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"]
Oct 31 07:53:14.136206 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.136145    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4bf7c-0fab-447f-9683-bc8930989fef-proxy-tls\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.136395 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.136228    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.136395 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.136288    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l7z4\" (UniqueName: \"kubernetes.io/projected/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-api-access-2l7z4\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.237623 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.237583    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4bf7c-0fab-447f-9683-bc8930989fef-proxy-tls\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.237795 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.237663    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.237795 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.237712    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7z4\" (UniqueName: \"kubernetes.io/projected/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-api-access-2l7z4\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.238396 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.238373    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.240252 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.240229    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4bf7c-0fab-447f-9683-bc8930989fef-proxy-tls\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.245402 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.245376    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l7z4\" (UniqueName: \"kubernetes.io/projected/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-api-access-2l7z4\") pod \"message-dumper-predictor-7f975599d-c7bp8\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") " pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.400173 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.400075    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:14.669839 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.669769    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"]
Oct 31 07:53:14.672119 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:53:14.672091    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4bf7c_0fab_447f_9683_bc8930989fef.slice/crio-42bf20d9884fcdf9a9fe8968ba2abce6aae03afe0497787d0b6c84c4e68cd5f4 WatchSource:0}: Error finding container 42bf20d9884fcdf9a9fe8968ba2abce6aae03afe0497787d0b6c84c4e68cd5f4: Status 404 returned error can't find the container with id 42bf20d9884fcdf9a9fe8968ba2abce6aae03afe0497787d0b6c84c4e68cd5f4
Oct 31 07:53:14.673987 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:14.673971    4913 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:53:15.616046 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:15.616013    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" event={"ID":"45b4bf7c-0fab-447f-9683-bc8930989fef","Type":"ContainerStarted","Data":"42bf20d9884fcdf9a9fe8968ba2abce6aae03afe0497787d0b6c84c4e68cd5f4"}
Oct 31 07:53:16.627573 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:16.627533    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" event={"ID":"45b4bf7c-0fab-447f-9683-bc8930989fef","Type":"ContainerStarted","Data":"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01"}
Oct 31 07:53:16.627999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:16.627620    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" event={"ID":"45b4bf7c-0fab-447f-9683-bc8930989fef","Type":"ContainerStarted","Data":"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77"}
Oct 31 07:53:16.627999 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:16.627682    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:16.643936 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:16.643882    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" podStartSLOduration=1.500059474 podStartE2EDuration="2.643867617s" totalImagesPullingTime="1.143808143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:53:16.643344739 +0000 UTC m=+1445.333113037" watchObservedRunningTime="2026-10-31 07:53:16.643867617 +0000 UTC m=+1445.333635909"
Oct 31 07:53:17.632022 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:17.631992    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:17.633780 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:17.633753    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:19.399659 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:19.399615    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:53:24.644949 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:24.644920    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:53:34.177541 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.177493    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"]
Oct 31 07:53:34.181479 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.181453    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.185558 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.185514    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Oct 31 07:53:34.185558 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.185541    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:53:34.185558 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.185527    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 07:53:34.186361 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.186342    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-logger-predictor-serving-cert\""
Oct 31 07:53:34.191258 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.191235    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"]
Oct 31 07:53:34.306768 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.306726    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2538-11d2-4351-99c0-78050881e301-proxy-tls\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.306768 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.306769    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vws88\" (UniqueName: \"kubernetes.io/projected/8c0a2538-11d2-4351-99c0-78050881e301-kube-api-access-vws88\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.306985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.306788    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-cabundle-cert\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.306985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.306922    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.306985 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.306962    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c0a2538-11d2-4351-99c0-78050881e301-kserve-provision-location\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.408312 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.408269    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2538-11d2-4351-99c0-78050881e301-proxy-tls\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.408312 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.408316    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vws88\" (UniqueName: \"kubernetes.io/projected/8c0a2538-11d2-4351-99c0-78050881e301-kube-api-access-vws88\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.408562 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.408335    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-cabundle-cert\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.408562 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.408380    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.408562 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.408416    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c0a2538-11d2-4351-99c0-78050881e301-kserve-provision-location\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.408949 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.408923    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c0a2538-11d2-4351-99c0-78050881e301-kserve-provision-location\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.409131 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.409112    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-cabundle-cert\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.409174 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.409142    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.410869 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.410850    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2538-11d2-4351-99c0-78050881e301-proxy-tls\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.416037 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.416008    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vws88\" (UniqueName: \"kubernetes.io/projected/8c0a2538-11d2-4351-99c0-78050881e301-kube-api-access-vws88\") pod \"isvc-logger-predictor-847654cb89-tgwvv\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") " pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.503048 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.502955    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:34.774106 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:34.774023    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"]
Oct 31 07:53:34.777063 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:53:34.777033    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0a2538_11d2_4351_99c0_78050881e301.slice/crio-673472d47b08949cb9e46645ec170c290b27b1f394425038552f18712e9e1d2e WatchSource:0}: Error finding container 673472d47b08949cb9e46645ec170c290b27b1f394425038552f18712e9e1d2e: Status 404 returned error can't find the container with id 673472d47b08949cb9e46645ec170c290b27b1f394425038552f18712e9e1d2e
Oct 31 07:53:35.709844 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:35.709801    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerStarted","Data":"92084942093cb0722482a9864d7cd3245c9f0b7541b7b55023741236d01aaa48"}
Oct 31 07:53:35.710271 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:35.709867    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerStarted","Data":"673472d47b08949cb9e46645ec170c290b27b1f394425038552f18712e9e1d2e"}
Oct 31 07:53:39.727371 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:39.727330    4913 generic.go:356] "Generic (PLEG): container finished" podID="8c0a2538-11d2-4351-99c0-78050881e301" containerID="92084942093cb0722482a9864d7cd3245c9f0b7541b7b55023741236d01aaa48" exitCode=0
Oct 31 07:53:39.727778 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:39.727403    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerDied","Data":"92084942093cb0722482a9864d7cd3245c9f0b7541b7b55023741236d01aaa48"}
Oct 31 07:53:40.734239 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:40.734203    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerStarted","Data":"47d18222756dba9acc71c7f4be3b1797398bda9a4f582f68bfb6236b5b7e024a"}
Oct 31 07:53:40.734631 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:40.734298    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerStarted","Data":"d5ba356dc3cec2d738be4692a3ce0dffa274b384d5e5269a5899aab9b233c1a2"}
Oct 31 07:53:40.734631 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:40.734324    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerStarted","Data":"ac91957d603c3fcc15e4b66436ab3632dbe033d1a52a81e059e49d3c60354e96"}
Oct 31 07:53:40.734631 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:40.734612    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:40.734790 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:40.734771    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:40.736097 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:40.736069    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:53:40.755965 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:40.755922    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podStartSLOduration=2.636147742 podStartE2EDuration="6.755910191s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.119762449s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:53:40.754529891 +0000 UTC m=+1469.444298196" watchObservedRunningTime="2026-10-31 07:53:40.755910191 +0000 UTC m=+1469.445678483"
Oct 31 07:53:41.738611 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:41.738576    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:41.739088 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:41.738692    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:53:41.739893 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:41.739870    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:53:42.742638 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:42.742600    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:53:42.743088 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:42.742997    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:53:47.747149 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:47.747120    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:53:47.747761 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:47.747719    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:53:47.748125 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:47.748103    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:53:49.399644 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:53:49.399603    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:53:57.748171 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:57.748090    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:53:57.748569 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:53:57.748537    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:54:07.748365 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:07.748322    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:54:07.748934 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:07.748867    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:54:11.542070 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:11.542027    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:54:11.551040 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:11.551014    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:54:17.748498 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:17.748453    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:54:17.748965 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:17.748936    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:54:19.402967 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:54:19.402912    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:54:27.748077 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:27.748034    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:54:27.748577 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:27.748554    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:54:37.748313 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:37.748262    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:54:37.748732 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:37.748666    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:54:47.747865 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:47.747826    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:54:47.748393 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:47.748357    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:54:49.399297 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:54:49.399262    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:54:51.416203 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:51.416136    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:54:51.416793 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:54:51.416607    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:55:01.418919 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:01.418886    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:55:01.419397 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:01.418980    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:55:09.933682 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:09.933645    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-13001498cc_message-dumper-predictor-7f975599d-c7bp8_45b4bf7c-0fab-447f-9683-bc8930989fef/kserve-container/0.log"
Oct 31 07:55:10.122034 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.121999    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"]
Oct 31 07:55:10.122393 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.122344    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" containerID="cri-o://ac91957d603c3fcc15e4b66436ab3632dbe033d1a52a81e059e49d3c60354e96" gracePeriod=30
Oct 31 07:55:10.122774 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.122704    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy" containerID="cri-o://d5ba356dc3cec2d738be4692a3ce0dffa274b384d5e5269a5899aab9b233c1a2" gracePeriod=30
Oct 31 07:55:10.122774 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.122735    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" containerID="cri-o://47d18222756dba9acc71c7f4be3b1797398bda9a4f582f68bfb6236b5b7e024a" gracePeriod=30
Oct 31 07:55:10.156203 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.156155    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"]
Oct 31 07:55:10.156481 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.156452    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerName="kserve-container" containerID="cri-o://ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77" gracePeriod=30
Oct 31 07:55:10.156843 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.156817    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerName="kube-rbac-proxy" containerID="cri-o://a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01" gracePeriod=30
Oct 31 07:55:10.404484 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.404458    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:55:10.497037 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.497003    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-api-access-2l7z4\" (UniqueName: \"kubernetes.io/projected/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-api-access-2l7z4\") pod \"45b4bf7c-0fab-447f-9683-bc8930989fef\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") "
Oct 31 07:55:10.497205 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.497057    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45b4bf7c-0fab-447f-9683-bc8930989fef-proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4bf7c-0fab-447f-9683-bc8930989fef-proxy-tls\") pod \"45b4bf7c-0fab-447f-9683-bc8930989fef\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") "
Oct 31 07:55:10.497205 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.497130    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-rbac-proxy-sar-config\") pod \"45b4bf7c-0fab-447f-9683-bc8930989fef\" (UID: \"45b4bf7c-0fab-447f-9683-bc8930989fef\") "
Oct 31 07:55:10.497511 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.497485    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-rbac-proxy-sar-config" pod "45b4bf7c-0fab-447f-9683-bc8930989fef" (UID: "45b4bf7c-0fab-447f-9683-bc8930989fef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:10.499132 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.499105    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-api-access-2l7z4" pod "45b4bf7c-0fab-447f-9683-bc8930989fef" (UID: "45b4bf7c-0fab-447f-9683-bc8930989fef"). InnerVolumeSpecName "kube-api-access-2l7z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:55:10.499263 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.499113    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b4bf7c-0fab-447f-9683-bc8930989fef-proxy-tls" pod "45b4bf7c-0fab-447f-9683-bc8930989fef" (UID: "45b4bf7c-0fab-447f-9683-bc8930989fef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:55:10.598206 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.598106    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45b4bf7c-0fab-447f-9683-bc8930989fef-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:10.598206 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.598137    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:10.598206 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:10.598149    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l7z4\" (UniqueName: \"kubernetes.io/projected/45b4bf7c-0fab-447f-9683-bc8930989fef-kube-api-access-2l7z4\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:11.112838 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.112805    4913 generic.go:356] "Generic (PLEG): container finished" podID="8c0a2538-11d2-4351-99c0-78050881e301" containerID="d5ba356dc3cec2d738be4692a3ce0dffa274b384d5e5269a5899aab9b233c1a2" exitCode=0
Oct 31 07:55:11.113265 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.112878    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerDied","Data":"d5ba356dc3cec2d738be4692a3ce0dffa274b384d5e5269a5899aab9b233c1a2"}
Oct 31 07:55:11.120302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.120267    4913 generic.go:356] "Generic (PLEG): container finished" podID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerID="a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01" exitCode=0
Oct 31 07:55:11.120302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.120300    4913 generic.go:356] "Generic (PLEG): container finished" podID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerID="ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77" exitCode=2
Oct 31 07:55:11.120477 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.120344    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"
Oct 31 07:55:11.120477 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.120349    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" event={"ID":"45b4bf7c-0fab-447f-9683-bc8930989fef","Type":"ContainerDied","Data":"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01"}
Oct 31 07:55:11.120477 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.120433    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" event={"ID":"45b4bf7c-0fab-447f-9683-bc8930989fef","Type":"ContainerDied","Data":"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77"}
Oct 31 07:55:11.120477 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.120457    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8" event={"ID":"45b4bf7c-0fab-447f-9683-bc8930989fef","Type":"ContainerDied","Data":"42bf20d9884fcdf9a9fe8968ba2abce6aae03afe0497787d0b6c84c4e68cd5f4"}
Oct 31 07:55:11.120614 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.120483    4913 scope.go:122] "RemoveContainer" containerID="a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01"
Oct 31 07:55:11.132219 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.132069    4913 scope.go:122] "RemoveContainer" containerID="ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77"
Oct 31 07:55:11.141926 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.141909    4913 scope.go:122] "RemoveContainer" containerID="a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01"
Oct 31 07:55:11.142239 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:55:11.142207    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01\": container with ID starting with a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01 not found: ID does not exist" containerID="a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01"
Oct 31 07:55:11.142321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.142249    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01"} err="failed to get container status \"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01\": rpc error: code = NotFound desc = could not find container \"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01\": container with ID starting with a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01 not found: ID does not exist"
Oct 31 07:55:11.142321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.142271    4913 scope.go:122] "RemoveContainer" containerID="ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77"
Oct 31 07:55:11.142537 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:55:11.142519    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77\": container with ID starting with ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77 not found: ID does not exist" containerID="ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77"
Oct 31 07:55:11.142574 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.142542    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77"} err="failed to get container status \"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77\": rpc error: code = NotFound desc = could not find container \"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77\": container with ID starting with ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77 not found: ID does not exist"
Oct 31 07:55:11.142574 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.142557    4913 scope.go:122] "RemoveContainer" containerID="a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01"
Oct 31 07:55:11.142761 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.142741    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01"} err="failed to get container status \"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01\": rpc error: code = NotFound desc = could not find container \"a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01\": container with ID starting with a48eb4b3b3e83a232a266b3174feaa2213ada5105d8481639452f1f9a3b9da01 not found: ID does not exist"
Oct 31 07:55:11.142805 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.142763    4913 scope.go:122] "RemoveContainer" containerID="ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77"
Oct 31 07:55:11.142958 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.142936    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77"} err="failed to get container status \"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77\": rpc error: code = NotFound desc = could not find container \"ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77\": container with ID starting with ccd452a8b69a9718860e8a9edcc8912566e3343ffccf7fef05761888dfe6ed77 not found: ID does not exist"
Oct 31 07:55:11.149941 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.149917    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"]
Oct 31 07:55:11.153423 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.153401    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/message-dumper-predictor-7f975599d-c7bp8"]
Oct 31 07:55:11.416316 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.416221    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:55:11.416494 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.416457    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:55:11.419860 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:11.419833    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" path="/var/lib/kubelet/pods/45b4bf7c-0fab-447f-9683-bc8930989fef/volumes"
Oct 31 07:55:12.743073 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:12.743030    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 31 07:55:15.140586 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:15.140559    4913 generic.go:356] "Generic (PLEG): container finished" podID="8c0a2538-11d2-4351-99c0-78050881e301" containerID="ac91957d603c3fcc15e4b66436ab3632dbe033d1a52a81e059e49d3c60354e96" exitCode=0
Oct 31 07:55:15.140912 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:15.140606    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerDied","Data":"ac91957d603c3fcc15e4b66436ab3632dbe033d1a52a81e059e49d3c60354e96"}
Oct 31 07:55:17.743618 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:17.743573    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 31 07:55:19.399802 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:55:19.399765    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:55:21.416490 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:21.416443    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:55:21.416974 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:21.416830    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:55:22.743760 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:22.743701    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 31 07:55:22.744274 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:22.743850    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:55:27.743098 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:27.743052    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 31 07:55:31.416111 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:31.416074    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Oct 31 07:55:31.416651 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:31.416401    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:55:31.424104 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:31.420118    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:55:31.424104 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:31.420326    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:55:32.743351 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:32.743309    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 31 07:55:37.743369 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:37.743302    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Oct 31 07:55:40.248692 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.248655    4913 generic.go:356] "Generic (PLEG): container finished" podID="8c0a2538-11d2-4351-99c0-78050881e301" containerID="47d18222756dba9acc71c7f4be3b1797398bda9a4f582f68bfb6236b5b7e024a" exitCode=0
Oct 31 07:55:40.249055 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.248724    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerDied","Data":"47d18222756dba9acc71c7f4be3b1797398bda9a4f582f68bfb6236b5b7e024a"}
Oct 31 07:55:40.275473 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.275451    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:55:40.453821 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.453792    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0a2538-11d2-4351-99c0-78050881e301-kube-api-access-vws88\" (UniqueName: \"kubernetes.io/projected/8c0a2538-11d2-4351-99c0-78050881e301-kube-api-access-vws88\") pod \"8c0a2538-11d2-4351-99c0-78050881e301\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") "
Oct 31 07:55:40.453973 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.453864    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c0a2538-11d2-4351-99c0-78050881e301-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c0a2538-11d2-4351-99c0-78050881e301-kserve-provision-location\") pod \"8c0a2538-11d2-4351-99c0-78050881e301\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") "
Oct 31 07:55:40.453973 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.453892    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c0a2538-11d2-4351-99c0-78050881e301-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2538-11d2-4351-99c0-78050881e301-proxy-tls\") pod \"8c0a2538-11d2-4351-99c0-78050881e301\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") "
Oct 31 07:55:40.453973 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.453928    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-cabundle-cert\") pod \"8c0a2538-11d2-4351-99c0-78050881e301\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") "
Oct 31 07:55:40.453973 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.453949    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-kube-rbac-proxy-sar-config\") pod \"8c0a2538-11d2-4351-99c0-78050881e301\" (UID: \"8c0a2538-11d2-4351-99c0-78050881e301\") "
Oct 31 07:55:40.454327 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.454220    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0a2538-11d2-4351-99c0-78050881e301-kserve-provision-location" pod "8c0a2538-11d2-4351-99c0-78050881e301" (UID: "8c0a2538-11d2-4351-99c0-78050881e301"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:55:40.454463 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.454349    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-cabundle-cert" pod "8c0a2538-11d2-4351-99c0-78050881e301" (UID: "8c0a2538-11d2-4351-99c0-78050881e301"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:40.454463 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.454377    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-kube-rbac-proxy-sar-config" pod "8c0a2538-11d2-4351-99c0-78050881e301" (UID: "8c0a2538-11d2-4351-99c0-78050881e301"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:40.455994 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.455975    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0a2538-11d2-4351-99c0-78050881e301-kube-api-access-vws88" pod "8c0a2538-11d2-4351-99c0-78050881e301" (UID: "8c0a2538-11d2-4351-99c0-78050881e301"). InnerVolumeSpecName "kube-api-access-vws88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:55:40.455994 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.455981    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0a2538-11d2-4351-99c0-78050881e301-proxy-tls" pod "8c0a2538-11d2-4351-99c0-78050881e301" (UID: "8c0a2538-11d2-4351-99c0-78050881e301"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:55:40.554566 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.554529    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vws88\" (UniqueName: \"kubernetes.io/projected/8c0a2538-11d2-4351-99c0-78050881e301-kube-api-access-vws88\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:40.554566 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.554558    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c0a2538-11d2-4351-99c0-78050881e301-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:40.554566 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.554570    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0a2538-11d2-4351-99c0-78050881e301-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:40.554801 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.554579    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:40.554801 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:40.554590    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c0a2538-11d2-4351-99c0-78050881e301-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:55:41.256340 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.256303    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv" event={"ID":"8c0a2538-11d2-4351-99c0-78050881e301","Type":"ContainerDied","Data":"673472d47b08949cb9e46645ec170c290b27b1f394425038552f18712e9e1d2e"}
Oct 31 07:55:41.256340 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.256343    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"
Oct 31 07:55:41.256830 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.256375    4913 scope.go:122] "RemoveContainer" containerID="47d18222756dba9acc71c7f4be3b1797398bda9a4f582f68bfb6236b5b7e024a"
Oct 31 07:55:41.265644 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.265625    4913 scope.go:122] "RemoveContainer" containerID="d5ba356dc3cec2d738be4692a3ce0dffa274b384d5e5269a5899aab9b233c1a2"
Oct 31 07:55:41.273053 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.273034    4913 scope.go:122] "RemoveContainer" containerID="ac91957d603c3fcc15e4b66436ab3632dbe033d1a52a81e059e49d3c60354e96"
Oct 31 07:55:41.276981 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.276959    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"]
Oct 31 07:55:41.281150 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.281128    4913 scope.go:122] "RemoveContainer" containerID="92084942093cb0722482a9864d7cd3245c9f0b7541b7b55023741236d01aaa48"
Oct 31 07:55:41.284405 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.284383    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-logger-predictor-847654cb89-tgwvv"]
Oct 31 07:55:41.419157 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:41.419125    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0a2538-11d2-4351-99c0-78050881e301" path="/var/lib/kubelet/pods/8c0a2538-11d2-4351-99c0-78050881e301/volumes"
Oct 31 07:55:42.056840 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.056804    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"]
Oct 31 07:55:42.057145 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057132    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="storage-initializer"
Oct 31 07:55:42.057220 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057147    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="storage-initializer"
Oct 31 07:55:42.057220 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057158    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy"
Oct 31 07:55:42.057220 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057163    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy"
Oct 31 07:55:42.057220 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057177    4913 cpu_manager.go:414] "containerMap: removing container" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerName="kserve-container"
Oct 31 07:55:42.057220 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057208    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerName="kserve-container"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057225    4913 cpu_manager.go:414] "containerMap: removing container" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerName="kube-rbac-proxy"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057230    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerName="kube-rbac-proxy"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057246    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057251    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057262    4913 cpu_manager.go:414] "containerMap: removing container" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057267    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057325    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerName="kserve-container"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057334    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b4bf7c-0fab-447f-9683-bc8930989fef" containerName="kube-rbac-proxy"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057341    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kserve-container"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057348    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="agent"
Oct 31 07:55:42.057379 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.057354    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0a2538-11d2-4351-99c0-78050881e301" containerName="kube-rbac-proxy"
Oct 31 07:55:42.061912 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.061890    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.065708 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.065664    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:55:42.065864 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.065781    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:55:42.065864 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.065781    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Oct 31 07:55:42.067628 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.067613    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:55:42.069654 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.069633    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"]
Oct 31 07:55:42.166159 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.166120    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 07:55:42.166371 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.166262    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:55:42.166371 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.166323    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Oct 31 07:55:42.166371 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.166344    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.166495 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.166402    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.166495 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.166451    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5bm\" (UniqueName: \"kubernetes.io/projected/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-api-access-jt5bm\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.166569 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.166504    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.166569 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.166528    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.267154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.267116    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.267154 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.267154    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.267734 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.267223    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.267734 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.267266    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.267734 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.267281    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5bm\" (UniqueName: \"kubernetes.io/projected/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-api-access-jt5bm\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.267734 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.267574    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.267896 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.267861    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.267935 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.267892    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.269767 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.269750    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-proxy-tls\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.274824 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.274798    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5bm\" (UniqueName: \"kubernetes.io/projected/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-api-access-jt5bm\") pod \"isvc-autogluon-v1-predictor-755fbbb795-h8d7p\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.381010 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.380927    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:55:42.651422 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:42.651349    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"]
Oct 31 07:55:42.654360 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:55:42.654333    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd5dfab_0beb_4a12_a1c4_1424316f6be5.slice/crio-e169e4ded906496e59c83b33066b3aad01fd4dcd1ea82f014c60061c8e2435ae WatchSource:0}: Error finding container e169e4ded906496e59c83b33066b3aad01fd4dcd1ea82f014c60061c8e2435ae: Status 404 returned error can't find the container with id e169e4ded906496e59c83b33066b3aad01fd4dcd1ea82f014c60061c8e2435ae
Oct 31 07:55:43.268400 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:43.268359    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" event={"ID":"bbd5dfab-0beb-4a12-a1c4-1424316f6be5","Type":"ContainerStarted","Data":"776a7242cbcd4ed64fa5c74d919d163d9bc5113d2402a770aa58f4b7061e645f"}
Oct 31 07:55:43.268888 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:43.268444    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" event={"ID":"bbd5dfab-0beb-4a12-a1c4-1424316f6be5","Type":"ContainerStarted","Data":"e169e4ded906496e59c83b33066b3aad01fd4dcd1ea82f014c60061c8e2435ae"}
Oct 31 07:55:49.293311 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:49.293278    4913 generic.go:356] "Generic (PLEG): container finished" podID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerID="776a7242cbcd4ed64fa5c74d919d163d9bc5113d2402a770aa58f4b7061e645f" exitCode=0
Oct 31 07:55:49.293689 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:55:49.293353    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" event={"ID":"bbd5dfab-0beb-4a12-a1c4-1424316f6be5","Type":"ContainerDied","Data":"776a7242cbcd4ed64fa5c74d919d163d9bc5113d2402a770aa58f4b7061e645f"}
Oct 31 07:55:49.400208 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:55:49.400149    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:56:02.148064 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.148030    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"]
Oct 31 07:56:02.158051 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.158023    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.161008 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.160980    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"]
Oct 31 07:56:02.164231 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.164177    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:56:02.164366 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.164279    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:56:02.165290 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.165050    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Oct 31 07:56:02.165290 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.165113    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:56:02.165290 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.165069    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:56:02.165290 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.165214    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:56:02.165290 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.165260    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Oct 31 07:56:02.231362 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.231327    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50513306-be8c-4e7b-b06e-b5177d8d6674-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.231542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.231383    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.231542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.231438    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50513306-be8c-4e7b-b06e-b5177d8d6674-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.231542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.231459    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.231542 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.231484    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95fbf\" (UniqueName: \"kubernetes.io/projected/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-api-access-95fbf\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.331893 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.331857    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50513306-be8c-4e7b-b06e-b5177d8d6674-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.331893 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.331892    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.332118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.331919    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95fbf\" (UniqueName: \"kubernetes.io/projected/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-api-access-95fbf\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.332118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.331974    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50513306-be8c-4e7b-b06e-b5177d8d6674-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.332118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.332025    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.332655 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.332602    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50513306-be8c-4e7b-b06e-b5177d8d6674-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.332782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.332729    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.332782 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.332741    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.334585 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.334566    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50513306-be8c-4e7b-b06e-b5177d8d6674-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.340532 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.340510    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95fbf\" (UniqueName: \"kubernetes.io/projected/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-api-access-95fbf\") pod \"isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.483802 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.483763    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:02.784844 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:02.784752    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"]
Oct 31 07:56:03.370011 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:03.369966    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" event={"ID":"50513306-be8c-4e7b-b06e-b5177d8d6674","Type":"ContainerStarted","Data":"60abe59b5dbd35d0caf205284a3800914edaff260bae323030f1f64920e6e7b2"}
Oct 31 07:56:03.370483 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:03.370044    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" event={"ID":"50513306-be8c-4e7b-b06e-b5177d8d6674","Type":"ContainerStarted","Data":"0eed56545c9d5f819fa74e5a993a1423bc87d6cbdb68e2b74e2e5d65d0308793"}
Oct 31 07:56:09.406794 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:09.406725    4913 generic.go:356] "Generic (PLEG): container finished" podID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerID="60abe59b5dbd35d0caf205284a3800914edaff260bae323030f1f64920e6e7b2" exitCode=0
Oct 31 07:56:09.407309 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:09.406801    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" event={"ID":"50513306-be8c-4e7b-b06e-b5177d8d6674","Type":"ContainerDied","Data":"60abe59b5dbd35d0caf205284a3800914edaff260bae323030f1f64920e6e7b2"}
Oct 31 07:56:10.415586 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:10.415550    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" event={"ID":"50513306-be8c-4e7b-b06e-b5177d8d6674","Type":"ContainerStarted","Data":"c20737c719dbaf46386e6816ef65fe53acc7ea5daf171494410f304e4ac1f4a1"}
Oct 31 07:56:10.416203 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:10.415629    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" event={"ID":"50513306-be8c-4e7b-b06e-b5177d8d6674","Type":"ContainerStarted","Data":"26a1ea955801985b409fea9ee179d290b881c31ca7bc3d4e0083499c31eda771"}
Oct 31 07:56:10.416203 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:10.415844    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:10.435489 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:10.435426    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" podStartSLOduration=2.03944642 podStartE2EDuration="8.435405789s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.395959369s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:56:10.433977544 +0000 UTC m=+1619.123745836" watchObservedRunningTime="2026-10-31 07:56:10.435405789 +0000 UTC m=+1619.125174082"
Oct 31 07:56:11.421962 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:11.421929    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:11.423127 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:11.423100    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 31 07:56:12.427340 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:12.427266    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 31 07:56:17.433302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:17.433268    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:17.434225 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:17.434177    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 31 07:56:19.409173 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:56:19.409131    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:56:27.434475 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:27.434432    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Oct 31 07:56:37.435645 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:37.435613    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:45.548508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:45.548250    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"]
Oct 31 07:56:45.549134 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:45.548521    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kserve-container" containerID="cri-o://26a1ea955801985b409fea9ee179d290b881c31ca7bc3d4e0083499c31eda771" gracePeriod=30
Oct 31 07:56:45.549134 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:45.548765    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kube-rbac-proxy" containerID="cri-o://c20737c719dbaf46386e6816ef65fe53acc7ea5daf171494410f304e4ac1f4a1" gracePeriod=30
Oct 31 07:56:46.605138 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.605102    4913 generic.go:356] "Generic (PLEG): container finished" podID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerID="c20737c719dbaf46386e6816ef65fe53acc7ea5daf171494410f304e4ac1f4a1" exitCode=0
Oct 31 07:56:46.605579 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.605140    4913 generic.go:356] "Generic (PLEG): container finished" podID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerID="26a1ea955801985b409fea9ee179d290b881c31ca7bc3d4e0083499c31eda771" exitCode=0
Oct 31 07:56:46.605579 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.605202    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" event={"ID":"50513306-be8c-4e7b-b06e-b5177d8d6674","Type":"ContainerDied","Data":"c20737c719dbaf46386e6816ef65fe53acc7ea5daf171494410f304e4ac1f4a1"}
Oct 31 07:56:46.605579 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.605254    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" event={"ID":"50513306-be8c-4e7b-b06e-b5177d8d6674","Type":"ContainerDied","Data":"26a1ea955801985b409fea9ee179d290b881c31ca7bc3d4e0083499c31eda771"}
Oct 31 07:56:46.605579 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.605275    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b" event={"ID":"50513306-be8c-4e7b-b06e-b5177d8d6674","Type":"ContainerDied","Data":"0eed56545c9d5f819fa74e5a993a1423bc87d6cbdb68e2b74e2e5d65d0308793"}
Oct 31 07:56:46.605579 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.605294    4913 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eed56545c9d5f819fa74e5a993a1423bc87d6cbdb68e2b74e2e5d65d0308793"
Oct 31 07:56:46.618936 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.618860    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:46.649329 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.649294    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50513306-be8c-4e7b-b06e-b5177d8d6674-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50513306-be8c-4e7b-b06e-b5177d8d6674-kserve-provision-location\") pod \"50513306-be8c-4e7b-b06e-b5177d8d6674\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") "
Oct 31 07:56:46.649508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.649353    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-rbac-proxy-sar-config\") pod \"50513306-be8c-4e7b-b06e-b5177d8d6674\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") "
Oct 31 07:56:46.649508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.649427    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-cabundle-cert\") pod \"50513306-be8c-4e7b-b06e-b5177d8d6674\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") "
Oct 31 07:56:46.649508 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.649466    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50513306-be8c-4e7b-b06e-b5177d8d6674-proxy-tls\" (UniqueName: \"kubernetes.io/secret/50513306-be8c-4e7b-b06e-b5177d8d6674-proxy-tls\") pod \"50513306-be8c-4e7b-b06e-b5177d8d6674\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") "
Oct 31 07:56:46.649705 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.649556    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-api-access-95fbf\" (UniqueName: \"kubernetes.io/projected/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-api-access-95fbf\") pod \"50513306-be8c-4e7b-b06e-b5177d8d6674\" (UID: \"50513306-be8c-4e7b-b06e-b5177d8d6674\") "
Oct 31 07:56:46.649952 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.649913    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-cabundle-cert" pod "50513306-be8c-4e7b-b06e-b5177d8d6674" (UID: "50513306-be8c-4e7b-b06e-b5177d8d6674"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:56:46.650088 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.650057    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-rbac-proxy-sar-config" pod "50513306-be8c-4e7b-b06e-b5177d8d6674" (UID: "50513306-be8c-4e7b-b06e-b5177d8d6674"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:56:46.652118 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.652087    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50513306-be8c-4e7b-b06e-b5177d8d6674-proxy-tls" pod "50513306-be8c-4e7b-b06e-b5177d8d6674" (UID: "50513306-be8c-4e7b-b06e-b5177d8d6674"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:56:46.652320 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.652293    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-api-access-95fbf" pod "50513306-be8c-4e7b-b06e-b5177d8d6674" (UID: "50513306-be8c-4e7b-b06e-b5177d8d6674"). InnerVolumeSpecName "kube-api-access-95fbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:56:46.679413 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.679370    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50513306-be8c-4e7b-b06e-b5177d8d6674-kserve-provision-location" pod "50513306-be8c-4e7b-b06e-b5177d8d6674" (UID: "50513306-be8c-4e7b-b06e-b5177d8d6674"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:56:46.750755 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.750712    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50513306-be8c-4e7b-b06e-b5177d8d6674-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:56:46.750755 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.750753    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:56:46.750971 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.750769    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50513306-be8c-4e7b-b06e-b5177d8d6674-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:56:46.750971 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.750785    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50513306-be8c-4e7b-b06e-b5177d8d6674-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:56:46.750971 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:46.750798    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95fbf\" (UniqueName: \"kubernetes.io/projected/50513306-be8c-4e7b-b06e-b5177d8d6674-kube-api-access-95fbf\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:56:47.610893 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:47.610860    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"
Oct 31 07:56:47.629295 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:47.629258    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"]
Oct 31 07:56:47.632235 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:47.632206    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-runtime-predictor-55b46d9dd8-kx94b"]
Oct 31 07:56:49.403845 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:56:49.403793    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:56:49.422006 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:56:49.421968    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" path="/var/lib/kubelet/pods/50513306-be8c-4e7b-b06e-b5177d8d6674/volumes"
Oct 31 07:57:19.402369 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:57:19.402317    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:57:36.849809 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:36.849774    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" event={"ID":"bbd5dfab-0beb-4a12-a1c4-1424316f6be5","Type":"ContainerStarted","Data":"a1647e93542dca85c099979979eb97f8b568de6f2551bb7471ac4396916d8f89"}
Oct 31 07:57:36.850289 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:36.849833    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" event={"ID":"bbd5dfab-0beb-4a12-a1c4-1424316f6be5","Type":"ContainerStarted","Data":"38e7331f2fbc7b966cdb870f4812d832e9d63f8908b9d68ee27f3e461d2840e0"}
Oct 31 07:57:36.850289 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:36.849932    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:57:36.868959 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:36.868911    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" podStartSLOduration=2.303515901 podStartE2EDuration="1m54.868897957s" totalImagesPullingTime="1m46.525036175s" totalInitContainerRuntime="6.040345881s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:57:36.867315115 +0000 UTC m=+1705.557083407" watchObservedRunningTime="2026-10-31 07:57:36.868897957 +0000 UTC m=+1705.558666253"
Oct 31 07:57:37.854597 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:37.854561    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:57:37.855755 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:37.855721    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 31 07:57:38.858932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:38.858884    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 31 07:57:43.863344 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:43.863307    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:57:43.864128 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:43.863995    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Oct 31 07:57:49.403528 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:57:49.403487    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:57:53.864361 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:53.864321    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:57:58.818932 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.818896    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"]
Oct 31 07:57:58.819431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.819223    4913 cpu_manager.go:414] "containerMap: removing container" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kserve-container"
Oct 31 07:57:58.819431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.819234    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kserve-container"
Oct 31 07:57:58.819431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.819246    4913 cpu_manager.go:414] "containerMap: removing container" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="storage-initializer"
Oct 31 07:57:58.819431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.819252    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="storage-initializer"
Oct 31 07:57:58.819431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.819279    4913 cpu_manager.go:414] "containerMap: removing container" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kube-rbac-proxy"
Oct 31 07:57:58.819431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.819285    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kube-rbac-proxy"
Oct 31 07:57:58.819431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.819352    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kserve-container"
Oct 31 07:57:58.819431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.819365    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="50513306-be8c-4e7b-b06e-b5177d8d6674" containerName="kube-rbac-proxy"
Oct 31 07:57:58.962443 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.962404    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"]
Oct 31 07:57:58.962625 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.962537    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:58.967629 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.967603    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:57:58.968448 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.968426    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:57:58.968607 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.968468    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Oct 31 07:57:58.968607 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.968473    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:57:58.968607 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.968514    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Oct 31 07:57:58.968607 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.968525    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:57:58.968607 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:58.968568    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:57:59.086706 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.086612    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjnk\" (UniqueName: \"kubernetes.io/projected/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-api-access-cbjnk\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.086706 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.086657    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.086976 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.086760    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.086976 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.086802    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.086976 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.086830    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-proxy-tls\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.187325 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.187293    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.187528 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.187333    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-proxy-tls\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.187528 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.187381    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjnk\" (UniqueName: \"kubernetes.io/projected/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-api-access-cbjnk\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.187528 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.187399    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.187528 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.187429    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.187923 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.187900    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.188180 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.188157    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.188343 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.188326    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.190028 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.190005    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-proxy-tls\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.195152 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.195126    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjnk\" (UniqueName: \"kubernetes.io/projected/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-api-access-cbjnk\") pod \"isvc-sklearn-v2-predictor-746c97d6fd-ntn6r\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.336910 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.336837    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:57:59.636431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.636404    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"]
Oct 31 07:57:59.639103 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:57:59.639077    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ec3dcd_3ce2_4a35_9af5_fe1e3451b6a1.slice/crio-082a20758f726d579579105a0a751104b6204ddbdcdfaa4ba1d0f845fc1aaa44 WatchSource:0}: Error finding container 082a20758f726d579579105a0a751104b6204ddbdcdfaa4ba1d0f845fc1aaa44: Status 404 returned error can't find the container with id 082a20758f726d579579105a0a751104b6204ddbdcdfaa4ba1d0f845fc1aaa44
Oct 31 07:57:59.953842 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.953803    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" event={"ID":"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1","Type":"ContainerStarted","Data":"3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393"}
Oct 31 07:57:59.954265 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:57:59.953866    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" event={"ID":"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1","Type":"ContainerStarted","Data":"082a20758f726d579579105a0a751104b6204ddbdcdfaa4ba1d0f845fc1aaa44"}
Oct 31 07:58:03.970849 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:03.970817    4913 generic.go:356] "Generic (PLEG): container finished" podID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerID="3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393" exitCode=0
Oct 31 07:58:03.971275 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:03.970865    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" event={"ID":"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1","Type":"ContainerDied","Data":"3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393"}
Oct 31 07:58:04.978955 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:04.978917    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" event={"ID":"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1","Type":"ContainerStarted","Data":"3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653"}
Oct 31 07:58:04.979532 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:04.978982    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" event={"ID":"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1","Type":"ContainerStarted","Data":"c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da"}
Oct 31 07:58:04.979532 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:04.979303    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:58:04.979532 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:04.979436    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:58:04.980730 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:04.980705    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:58:04.998772 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:04.998730    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podStartSLOduration=3.084442316 podStartE2EDuration="6.998717781s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.914275465s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:58:04.9975622 +0000 UTC m=+1733.687330522" watchObservedRunningTime="2026-10-31 07:58:04.998717781 +0000 UTC m=+1733.688486116"
Oct 31 07:58:05.983431 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:05.983382    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:58:08.646539 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:08.646508    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"]
Oct 31 07:58:08.646943 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:08.646755    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kserve-container" containerID="cri-o://38e7331f2fbc7b966cdb870f4812d832e9d63f8908b9d68ee27f3e461d2840e0" gracePeriod=30
Oct 31 07:58:08.646943 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:08.646780    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kube-rbac-proxy" containerID="cri-o://a1647e93542dca85c099979979eb97f8b568de6f2551bb7471ac4396916d8f89" gracePeriod=30
Oct 31 07:58:08.859501 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:08.859455    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Oct 31 07:58:08.999615 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:08.999582    4913 generic.go:356] "Generic (PLEG): container finished" podID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerID="a1647e93542dca85c099979979eb97f8b568de6f2551bb7471ac4396916d8f89" exitCode=0
Oct 31 07:58:08.999803 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:08.999660    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" event={"ID":"bbd5dfab-0beb-4a12-a1c4-1424316f6be5","Type":"ContainerDied","Data":"a1647e93542dca85c099979979eb97f8b568de6f2551bb7471ac4396916d8f89"}
Oct 31 07:58:10.006823 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.006791    4913 generic.go:356] "Generic (PLEG): container finished" podID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerID="38e7331f2fbc7b966cdb870f4812d832e9d63f8908b9d68ee27f3e461d2840e0" exitCode=0
Oct 31 07:58:10.007222 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.006838    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" event={"ID":"bbd5dfab-0beb-4a12-a1c4-1424316f6be5","Type":"ContainerDied","Data":"38e7331f2fbc7b966cdb870f4812d832e9d63f8908b9d68ee27f3e461d2840e0"}
Oct 31 07:58:10.128552 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.128510    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:58:10.284922 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.284887    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-api-access-jt5bm\" (UniqueName: \"kubernetes.io/projected/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-api-access-jt5bm\") pod \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") "
Oct 31 07:58:10.285131 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.284945    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kserve-provision-location\") pod \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") "
Oct 31 07:58:10.285131 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.284972    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-rbac-proxy-sar-config\") pod \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") "
Oct 31 07:58:10.285131 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.285001    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-proxy-tls\") pod \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") "
Oct 31 07:58:10.285131 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.285051    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-cabundle-cert\") pod \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\" (UID: \"bbd5dfab-0beb-4a12-a1c4-1424316f6be5\") "
Oct 31 07:58:10.285407 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.285381    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-rbac-proxy-sar-config" pod "bbd5dfab-0beb-4a12-a1c4-1424316f6be5" (UID: "bbd5dfab-0beb-4a12-a1c4-1424316f6be5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:58:10.285519 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.285486    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-cabundle-cert" pod "bbd5dfab-0beb-4a12-a1c4-1424316f6be5" (UID: "bbd5dfab-0beb-4a12-a1c4-1424316f6be5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:58:10.287211 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.287167    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-api-access-jt5bm" pod "bbd5dfab-0beb-4a12-a1c4-1424316f6be5" (UID: "bbd5dfab-0beb-4a12-a1c4-1424316f6be5"). InnerVolumeSpecName "kube-api-access-jt5bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:58:10.287315 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.287170    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-proxy-tls" pod "bbd5dfab-0beb-4a12-a1c4-1424316f6be5" (UID: "bbd5dfab-0beb-4a12-a1c4-1424316f6be5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:58:10.289652 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.289622    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kserve-provision-location" pod "bbd5dfab-0beb-4a12-a1c4-1424316f6be5" (UID: "bbd5dfab-0beb-4a12-a1c4-1424316f6be5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:58:10.386321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.386287    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:58:10.386321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.386317    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:58:10.386321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.386329    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:58:10.386550 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.386339    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5bm\" (UniqueName: \"kubernetes.io/projected/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kube-api-access-jt5bm\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:58:10.386550 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.386348    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd5dfab-0beb-4a12-a1c4-1424316f6be5-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:58:10.987860 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.987827    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:58:10.988387 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:10.988356    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:58:11.012914 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:11.012885    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"
Oct 31 07:58:11.013419 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:11.012886    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p" event={"ID":"bbd5dfab-0beb-4a12-a1c4-1424316f6be5","Type":"ContainerDied","Data":"e169e4ded906496e59c83b33066b3aad01fd4dcd1ea82f014c60061c8e2435ae"}
Oct 31 07:58:11.013419 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:11.013025    4913 scope.go:122] "RemoveContainer" containerID="a1647e93542dca85c099979979eb97f8b568de6f2551bb7471ac4396916d8f89"
Oct 31 07:58:11.021709 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:11.021685    4913 scope.go:122] "RemoveContainer" containerID="38e7331f2fbc7b966cdb870f4812d832e9d63f8908b9d68ee27f3e461d2840e0"
Oct 31 07:58:11.029837 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:11.029813    4913 scope.go:122] "RemoveContainer" containerID="776a7242cbcd4ed64fa5c74d919d163d9bc5113d2402a770aa58f4b7061e645f"
Oct 31 07:58:11.034087 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:11.034062    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"]
Oct 31 07:58:11.035655 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:11.035634    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v1-predictor-755fbbb795-h8d7p"]
Oct 31 07:58:11.419510 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:11.419423    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" path="/var/lib/kubelet/pods/bbd5dfab-0beb-4a12-a1c4-1424316f6be5/volumes"
Oct 31 07:58:13.175886 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.175850    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"]
Oct 31 07:58:13.176302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.176237    4913 cpu_manager.go:414] "containerMap: removing container" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="storage-initializer"
Oct 31 07:58:13.176302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.176251    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="storage-initializer"
Oct 31 07:58:13.176302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.176275    4913 cpu_manager.go:414] "containerMap: removing container" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kserve-container"
Oct 31 07:58:13.176302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.176281    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kserve-container"
Oct 31 07:58:13.176302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.176292    4913 cpu_manager.go:414] "containerMap: removing container" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kube-rbac-proxy"
Oct 31 07:58:13.176302 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.176297    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kube-rbac-proxy"
Oct 31 07:58:13.176519 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.176362    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kserve-container"
Oct 31 07:58:13.176519 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.176372    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd5dfab-0beb-4a12-a1c4-1424316f6be5" containerName="kube-rbac-proxy"
Oct 31 07:58:13.180724 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.180704    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.184202 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.184171    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Oct 31 07:58:13.184325 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.184256    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Oct 31 07:58:13.184325 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.184275    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:58:13.184325 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.184274    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 07:58:13.184325 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.184317    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 07:58:13.185008 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.184994    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 07:58:13.186775 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.186758    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 07:58:13.188987 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.188962    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"]
Oct 31 07:58:13.308670 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.308626    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcswm\" (UniqueName: \"kubernetes.io/projected/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-api-access-wcswm\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.308839 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.308687    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.308839 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.308770    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.308839 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.308812    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0942427d-58e1-4a41-9cba-cb3f57915f3b-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.309001 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.308959    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0942427d-58e1-4a41-9cba-cb3f57915f3b-proxy-tls\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.410373 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.410336    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0942427d-58e1-4a41-9cba-cb3f57915f3b-proxy-tls\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.410570 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.410381    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcswm\" (UniqueName: \"kubernetes.io/projected/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-api-access-wcswm\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.410570 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.410421    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.410570 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.410443    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.410570 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.410472    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0942427d-58e1-4a41-9cba-cb3f57915f3b-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.411000 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.410974    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0942427d-58e1-4a41-9cba-cb3f57915f3b-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.411315 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.411294    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.411396 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.411298    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.412859 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.412828    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0942427d-58e1-4a41-9cba-cb3f57915f3b-proxy-tls\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.417808 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.417782    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcswm\" (UniqueName: \"kubernetes.io/projected/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-api-access-wcswm\") pod \"isvc-autogluon-v2-predictor-8594d8f789-qrm5w\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.497958 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.497868    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:13.774951 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:13.774863    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"]
Oct 31 07:58:13.777816 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:58:13.777785    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0942427d_58e1_4a41_9cba_cb3f57915f3b.slice/crio-c11e69a319b784e2ea516abf77cc91bf08efc47658418dfb0902692dc2897891 WatchSource:0}: Error finding container c11e69a319b784e2ea516abf77cc91bf08efc47658418dfb0902692dc2897891: Status 404 returned error can't find the container with id c11e69a319b784e2ea516abf77cc91bf08efc47658418dfb0902692dc2897891
Oct 31 07:58:14.029005 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:14.028927    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" event={"ID":"0942427d-58e1-4a41-9cba-cb3f57915f3b","Type":"ContainerStarted","Data":"543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2"}
Oct 31 07:58:14.029005 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:14.028987    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" event={"ID":"0942427d-58e1-4a41-9cba-cb3f57915f3b","Type":"ContainerStarted","Data":"c11e69a319b784e2ea516abf77cc91bf08efc47658418dfb0902692dc2897891"}
Oct 31 07:58:19.399949 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:58:19.399907    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:58:20.053978 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:20.053892    4913 generic.go:356] "Generic (PLEG): container finished" podID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerID="543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2" exitCode=0
Oct 31 07:58:20.054155 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:20.053969    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" event={"ID":"0942427d-58e1-4a41-9cba-cb3f57915f3b","Type":"ContainerDied","Data":"543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2"}
Oct 31 07:58:20.988491 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:20.988449    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:58:21.060868 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:21.060832    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" event={"ID":"0942427d-58e1-4a41-9cba-cb3f57915f3b","Type":"ContainerStarted","Data":"3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c"}
Oct 31 07:58:21.061055 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:21.060901    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" event={"ID":"0942427d-58e1-4a41-9cba-cb3f57915f3b","Type":"ContainerStarted","Data":"cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1"}
Oct 31 07:58:21.061127 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:21.061112    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:21.080314 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:21.080261    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" podStartSLOduration=2.228094117 podStartE2EDuration="8.080247563s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.852153446s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:58:21.078444761 +0000 UTC m=+1749.768213053" watchObservedRunningTime="2026-10-31 07:58:21.080247563 +0000 UTC m=+1749.770015856"
Oct 31 07:58:22.065666 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:22.065634    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:28.075748 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:28.075718    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:58:30.988689 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:30.988646    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:58:40.988629 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:40.988588    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:58:49.399223 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:58:49.399162    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:58:50.989065 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:58:50.989023    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:59:00.988816 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:00.988776    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:59:10.989045 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:10.989004    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:59:11.573449 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:11.573418    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:59:11.578303 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:11.578279    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 07:59:19.402789 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:59:19.402750    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:59:20.988441 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:20.988408    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:59:30.989165 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:30.989134    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:59:39.951369 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:39.951336    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"]
Oct 31 07:59:39.951787 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:39.951573    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" containerID="cri-o://c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da" gracePeriod=30
Oct 31 07:59:39.951787 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:39.951633    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kube-rbac-proxy" containerID="cri-o://3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653" gracePeriod=30
Oct 31 07:59:40.406593 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:40.406508    4913 generic.go:356] "Generic (PLEG): container finished" podID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerID="3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653" exitCode=0
Oct 31 07:59:40.406747 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:40.406584    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" event={"ID":"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1","Type":"ContainerDied","Data":"3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653"}
Oct 31 07:59:40.984368 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:40.984323    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Oct 31 07:59:40.988732 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:40.988702    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Oct 31 07:59:45.003490 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.003465    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:59:45.062522 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.062484    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-api-access-cbjnk\" (UniqueName: \"kubernetes.io/projected/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-api-access-cbjnk\") pod \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") "
Oct 31 07:59:45.062690 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.062531    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-cabundle-cert\") pod \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") "
Oct 31 07:59:45.062690 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.062570    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kserve-provision-location\") pod \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") "
Oct 31 07:59:45.062690 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.062639    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-proxy-tls\") pod \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") "
Oct 31 07:59:45.062840 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.062724    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-rbac-proxy-sar-config\") pod \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\" (UID: \"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1\") "
Oct 31 07:59:45.063080 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.063048    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-cabundle-cert" pod "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" (UID: "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:59:45.063080 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.062987    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kserve-provision-location" pod "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" (UID: "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:59:45.063327 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.063129    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-rbac-proxy-sar-config" pod "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" (UID: "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:59:45.064832 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.064811    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-proxy-tls" pod "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" (UID: "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:59:45.064907 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.064851    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-api-access-cbjnk" pod "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" (UID: "73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1"). InnerVolumeSpecName "kube-api-access-cbjnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:59:45.164321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.164234    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjnk\" (UniqueName: \"kubernetes.io/projected/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-api-access-cbjnk\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:59:45.164321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.164269    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:59:45.164321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.164280    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:59:45.164321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.164291    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:59:45.164321 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.164299    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 07:59:45.429218 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.429168    4913 generic.go:356] "Generic (PLEG): container finished" podID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerID="c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da" exitCode=0
Oct 31 07:59:45.429347 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.429253    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"
Oct 31 07:59:45.429347 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.429252    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" event={"ID":"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1","Type":"ContainerDied","Data":"c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da"}
Oct 31 07:59:45.429347 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.429333    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r" event={"ID":"73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1","Type":"ContainerDied","Data":"082a20758f726d579579105a0a751104b6204ddbdcdfaa4ba1d0f845fc1aaa44"}
Oct 31 07:59:45.429520 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.429373    4913 scope.go:122] "RemoveContainer" containerID="3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653"
Oct 31 07:59:45.437617 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.437601    4913 scope.go:122] "RemoveContainer" containerID="c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da"
Oct 31 07:59:45.445676 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.445651    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"]
Oct 31 07:59:45.445831 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.445812    4913 scope.go:122] "RemoveContainer" containerID="3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393"
Oct 31 07:59:45.449497 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.449475    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-predictor-746c97d6fd-ntn6r"]
Oct 31 07:59:45.453775 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.453753    4913 scope.go:122] "RemoveContainer" containerID="3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653"
Oct 31 07:59:45.454019 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:59:45.454001    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653\": container with ID starting with 3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653 not found: ID does not exist" containerID="3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653"
Oct 31 07:59:45.454069 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.454029    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653"} err="failed to get container status \"3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653\": rpc error: code = NotFound desc = could not find container \"3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653\": container with ID starting with 3b9d0684cdd8c6b82ca1c4665906f39d9631ae881f419b0d6d0efede6cca5653 not found: ID does not exist"
Oct 31 07:59:45.454069 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.454045    4913 scope.go:122] "RemoveContainer" containerID="c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da"
Oct 31 07:59:45.454287 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:59:45.454268    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da\": container with ID starting with c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da not found: ID does not exist" containerID="c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da"
Oct 31 07:59:45.454336 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.454295    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da"} err="failed to get container status \"c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da\": rpc error: code = NotFound desc = could not find container \"c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da\": container with ID starting with c5127bf2d3a88efbd3529ea7cf027101be24704ec6aac7c1a3fe2768df2535da not found: ID does not exist"
Oct 31 07:59:45.454336 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.454315    4913 scope.go:122] "RemoveContainer" containerID="3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393"
Oct 31 07:59:45.454530 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:59:45.454514    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393\": container with ID starting with 3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393 not found: ID does not exist" containerID="3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393"
Oct 31 07:59:45.454573 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:45.454534    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393"} err="failed to get container status \"3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393\": rpc error: code = NotFound desc = could not find container \"3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393\": container with ID starting with 3e902eba8bf665805db1da8b6e6a6fc13611c1b4edd32d4519e8722bb63eb393 not found: ID does not exist"
Oct 31 07:59:47.420350 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.420319    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" path="/var/lib/kubelet/pods/73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1/volumes"
Oct 31 07:59:47.455039 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455008    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"]
Oct 31 07:59:47.455363 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455350    4913 cpu_manager.go:414] "containerMap: removing container" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="storage-initializer"
Oct 31 07:59:47.455415 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455365    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="storage-initializer"
Oct 31 07:59:47.455415 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455376    4913 cpu_manager.go:414] "containerMap: removing container" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kube-rbac-proxy"
Oct 31 07:59:47.455415 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455382    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kube-rbac-proxy"
Oct 31 07:59:47.455415 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455393    4913 cpu_manager.go:414] "containerMap: removing container" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container"
Oct 31 07:59:47.455415 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455398    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container"
Oct 31 07:59:47.455571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455458    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kube-rbac-proxy"
Oct 31 07:59:47.455571 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.455468    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ec3dcd-3ce2-4a35-9af5-fe1e3451b6a1" containerName="kserve-container"
Oct 31 07:59:47.460279 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.460260    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.463341 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.463322    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 07:59:47.466384 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.466363    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Oct 31 07:59:47.469085 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.469067    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:59:47.469085 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.469079    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 07:59:47.471358 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.471333    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"]
Oct 31 07:59:47.564569 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.564541    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 07:59:47.564745 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.564591    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Oct 31 07:59:47.567257 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.567236    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 07:59:47.581942 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.581915    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdtc\" (UniqueName: \"kubernetes.io/projected/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-api-access-xbdtc\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.582101 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.581967    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.582101 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.582082    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.582248 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.582113    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.582248 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.582146    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.683566 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.683480    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.683566 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.683534    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.683807 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.683654    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.683807 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.683706    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.683807 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.683777    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdtc\" (UniqueName: \"kubernetes.io/projected/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-api-access-xbdtc\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.683962 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.683895    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.684413 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.684389    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.684491 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.684419    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.686022 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.686000    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.691712 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.691687    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdtc\" (UniqueName: \"kubernetes.io/projected/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-api-access-xbdtc\") pod \"isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:47.779667 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:47.779634    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:48.050177 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:48.050092    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"]
Oct 31 07:59:48.053128 ip-10-0-130-121 kubenswrapper[4913]: W1031 07:59:48.053098    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db35884_c1ee_43a8_8a06_e9e9f41df5b9.slice/crio-a4f5e17cded05191543557e7c66c2c55b68ff3a1b97815c3bc001cebc91875c2 WatchSource:0}: Error finding container a4f5e17cded05191543557e7c66c2c55b68ff3a1b97815c3bc001cebc91875c2: Status 404 returned error can't find the container with id a4f5e17cded05191543557e7c66c2c55b68ff3a1b97815c3bc001cebc91875c2
Oct 31 07:59:48.447008 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:48.446973    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" event={"ID":"1db35884-c1ee-43a8-8a06-e9e9f41df5b9","Type":"ContainerStarted","Data":"e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03"}
Oct 31 07:59:48.447417 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:48.447029    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" event={"ID":"1db35884-c1ee-43a8-8a06-e9e9f41df5b9","Type":"ContainerStarted","Data":"a4f5e17cded05191543557e7c66c2c55b68ff3a1b97815c3bc001cebc91875c2"}
Oct 31 07:59:49.400152 ip-10-0-130-121 kubenswrapper[4913]: E1031 07:59:49.400114    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 07:59:52.464854 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:52.464823    4913 generic.go:356] "Generic (PLEG): container finished" podID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerID="e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03" exitCode=0
Oct 31 07:59:52.465247 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:52.464881    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" event={"ID":"1db35884-c1ee-43a8-8a06-e9e9f41df5b9","Type":"ContainerDied","Data":"e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03"}
Oct 31 07:59:53.448730 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:53.448644    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 07:59:53.472835 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:53.472793    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" event={"ID":"1db35884-c1ee-43a8-8a06-e9e9f41df5b9","Type":"ContainerStarted","Data":"dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7"}
Oct 31 07:59:53.473331 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:53.472878    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" event={"ID":"1db35884-c1ee-43a8-8a06-e9e9f41df5b9","Type":"ContainerStarted","Data":"53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6"}
Oct 31 07:59:53.473331 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:53.473100    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:53.473331 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:53.473147    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:53.474866 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:53.474836    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 07:59:53.490879 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:53.490536    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podStartSLOduration=2.676642144 podStartE2EDuration="6.490517791s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.813875647s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:59:53.488879052 +0000 UTC m=+1842.178647370" watchObservedRunningTime="2026-10-31 07:59:53.490517791 +0000 UTC m=+1842.180286083"
Oct 31 07:59:54.477725 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:54.477687    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 07:59:59.482037 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:59.482008    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 07:59:59.482509 ip-10-0-130-121 kubenswrapper[4913]: I1031 07:59:59.482484    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:00:04.455352 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:04.455307    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.61:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:00:09.482563 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:09.482521    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:00:09.647741 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:09.647707    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"]
Oct 31 08:00:09.648056 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:09.648026    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kserve-container" containerID="cri-o://cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1" gracePeriod=30
Oct 31 08:00:09.648131 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:09.648077    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kube-rbac-proxy" containerID="cri-o://3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c" gracePeriod=30
Oct 31 08:00:10.547509 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:10.547469    4913 generic.go:356] "Generic (PLEG): container finished" podID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerID="3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c" exitCode=0
Oct 31 08:00:10.547913 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:10.547531    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" event={"ID":"0942427d-58e1-4a41-9cba-cb3f57915f3b","Type":"ContainerDied","Data":"3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c"}
Oct 31 08:00:11.196229 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.196204    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 08:00:11.198214 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.198169    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-api-access-wcswm\" (UniqueName: \"kubernetes.io/projected/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-api-access-wcswm\") pod \"0942427d-58e1-4a41-9cba-cb3f57915f3b\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") "
Oct 31 08:00:11.198352 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.198240    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-cabundle-cert\") pod \"0942427d-58e1-4a41-9cba-cb3f57915f3b\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") "
Oct 31 08:00:11.198352 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.198260    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0942427d-58e1-4a41-9cba-cb3f57915f3b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/0942427d-58e1-4a41-9cba-cb3f57915f3b-proxy-tls\") pod \"0942427d-58e1-4a41-9cba-cb3f57915f3b\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") "
Oct 31 08:00:11.198352 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.198291    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0942427d-58e1-4a41-9cba-cb3f57915f3b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0942427d-58e1-4a41-9cba-cb3f57915f3b-kserve-provision-location\") pod \"0942427d-58e1-4a41-9cba-cb3f57915f3b\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") "
Oct 31 08:00:11.198527 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.198496    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-rbac-proxy-sar-config\") pod \"0942427d-58e1-4a41-9cba-cb3f57915f3b\" (UID: \"0942427d-58e1-4a41-9cba-cb3f57915f3b\") "
Oct 31 08:00:11.198606 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.198581    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-cabundle-cert" pod "0942427d-58e1-4a41-9cba-cb3f57915f3b" (UID: "0942427d-58e1-4a41-9cba-cb3f57915f3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:00:11.198826 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.198804    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:00:11.198922 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.198841    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-rbac-proxy-sar-config" pod "0942427d-58e1-4a41-9cba-cb3f57915f3b" (UID: "0942427d-58e1-4a41-9cba-cb3f57915f3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:00:11.200423 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.200400    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0942427d-58e1-4a41-9cba-cb3f57915f3b-proxy-tls" pod "0942427d-58e1-4a41-9cba-cb3f57915f3b" (UID: "0942427d-58e1-4a41-9cba-cb3f57915f3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:00:11.200511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.200429    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-api-access-wcswm" pod "0942427d-58e1-4a41-9cba-cb3f57915f3b" (UID: "0942427d-58e1-4a41-9cba-cb3f57915f3b"). InnerVolumeSpecName "kube-api-access-wcswm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:00:11.204650 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.204622    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0942427d-58e1-4a41-9cba-cb3f57915f3b-kserve-provision-location" pod "0942427d-58e1-4a41-9cba-cb3f57915f3b" (UID: "0942427d-58e1-4a41-9cba-cb3f57915f3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:00:11.299854 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.299777    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcswm\" (UniqueName: \"kubernetes.io/projected/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-api-access-wcswm\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:00:11.299854 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.299804    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0942427d-58e1-4a41-9cba-cb3f57915f3b-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:00:11.299854 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.299815    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0942427d-58e1-4a41-9cba-cb3f57915f3b-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:00:11.299854 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.299825    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0942427d-58e1-4a41-9cba-cb3f57915f3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:00:11.553948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.553850    4913 generic.go:356] "Generic (PLEG): container finished" podID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerID="cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1" exitCode=0
Oct 31 08:00:11.553948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.553904    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"
Oct 31 08:00:11.554434 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.553901    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" event={"ID":"0942427d-58e1-4a41-9cba-cb3f57915f3b","Type":"ContainerDied","Data":"cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1"}
Oct 31 08:00:11.554434 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.554050    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w" event={"ID":"0942427d-58e1-4a41-9cba-cb3f57915f3b","Type":"ContainerDied","Data":"c11e69a319b784e2ea516abf77cc91bf08efc47658418dfb0902692dc2897891"}
Oct 31 08:00:11.554434 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.554085    4913 scope.go:122] "RemoveContainer" containerID="3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c"
Oct 31 08:00:11.562556 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.562538    4913 scope.go:122] "RemoveContainer" containerID="cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1"
Oct 31 08:00:11.572717 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.572692    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"]
Oct 31 08:00:11.573010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.572993    4913 scope.go:122] "RemoveContainer" containerID="543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2"
Oct 31 08:00:11.576405 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.576383    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-predictor-8594d8f789-qrm5w"]
Oct 31 08:00:11.580716 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.580695    4913 scope.go:122] "RemoveContainer" containerID="3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c"
Oct 31 08:00:11.580994 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:00:11.580975    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c\": container with ID starting with 3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c not found: ID does not exist" containerID="3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c"
Oct 31 08:00:11.581067 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.581007    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c"} err="failed to get container status \"3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c\": rpc error: code = NotFound desc = could not find container \"3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c\": container with ID starting with 3724fd19ddd834dcf5755d6e48c9d0deb942a7e4a2a8dedf96238e72bf68e86c not found: ID does not exist"
Oct 31 08:00:11.581067 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.581034    4913 scope.go:122] "RemoveContainer" containerID="cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1"
Oct 31 08:00:11.581319 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:00:11.581292    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1\": container with ID starting with cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1 not found: ID does not exist" containerID="cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1"
Oct 31 08:00:11.581366 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.581326    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1"} err="failed to get container status \"cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1\": rpc error: code = NotFound desc = could not find container \"cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1\": container with ID starting with cd140755e57501929dce69d4402e4af4fdcf536dffbb97dea36b1ff4f56940e1 not found: ID does not exist"
Oct 31 08:00:11.581366 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.581342    4913 scope.go:122] "RemoveContainer" containerID="543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2"
Oct 31 08:00:11.581590 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:00:11.581570    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2\": container with ID starting with 543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2 not found: ID does not exist" containerID="543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2"
Oct 31 08:00:11.581647 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:11.581599    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2"} err="failed to get container status \"543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2\": rpc error: code = NotFound desc = could not find container \"543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2\": container with ID starting with 543255ef53e2851a1f5d765399c5872e2059618dc22bd9a268e15e01c0a9d2c2 not found: ID does not exist"
Oct 31 08:00:12.117054 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117017    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"]
Oct 31 08:00:12.117406 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117391    4913 cpu_manager.go:414] "containerMap: removing container" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="storage-initializer"
Oct 31 08:00:12.117457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117407    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="storage-initializer"
Oct 31 08:00:12.117457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117417    4913 cpu_manager.go:414] "containerMap: removing container" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kserve-container"
Oct 31 08:00:12.117457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117422    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kserve-container"
Oct 31 08:00:12.117457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117438    4913 cpu_manager.go:414] "containerMap: removing container" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kube-rbac-proxy"
Oct 31 08:00:12.117457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117443    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kube-rbac-proxy"
Oct 31 08:00:12.117640 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117497    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kserve-container"
Oct 31 08:00:12.117640 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.117510    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" containerName="kube-rbac-proxy"
Oct 31 08:00:12.122333 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.122316    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.129217 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.129054    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Oct 31 08:00:12.129217 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.129124    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:00:12.129217 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.129128    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:00:12.129217 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.129130    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:00:12.129217 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.129170    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Oct 31 08:00:12.129539 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.129254    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:00:12.130308 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.130287    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:00:12.132093 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.131924    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"]
Oct 31 08:00:12.208820 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.208782    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fpj5\" (UniqueName: \"kubernetes.io/projected/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-api-access-8fpj5\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.208994 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.208829    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.208994 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.208924    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79e6087c-f70d-4e91-b0f3-c0e77026e72c-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.208994 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.208966    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.208994 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.208987    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.309718 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.309676    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.309936 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.309729    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.309936 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.309780    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fpj5\" (UniqueName: \"kubernetes.io/projected/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-api-access-8fpj5\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.309936 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.309915    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.310115 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.310010    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79e6087c-f70d-4e91-b0f3-c0e77026e72c-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.310235 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.310214    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.310530 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.310507    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.310683 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.310666    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.312461 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.312444    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79e6087c-f70d-4e91-b0f3-c0e77026e72c-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.317108 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.317087    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fpj5\" (UniqueName: \"kubernetes.io/projected/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-api-access-8fpj5\") pod \"isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.443089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.443051    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:12.717520 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:12.717492    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"]
Oct 31 08:00:12.719775 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:00:12.719733    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e6087c_f70d_4e91_b0f3_c0e77026e72c.slice/crio-da406a669d24af22bf285393fbe44c7d55f584ffe47f2c8317b865554abc5471 WatchSource:0}: Error finding container da406a669d24af22bf285393fbe44c7d55f584ffe47f2c8317b865554abc5471: Status 404 returned error can't find the container with id da406a669d24af22bf285393fbe44c7d55f584ffe47f2c8317b865554abc5471
Oct 31 08:00:13.419412 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:13.419382    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0942427d-58e1-4a41-9cba-cb3f57915f3b" path="/var/lib/kubelet/pods/0942427d-58e1-4a41-9cba-cb3f57915f3b/volumes"
Oct 31 08:00:13.563497 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:13.563459    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" event={"ID":"79e6087c-f70d-4e91-b0f3-c0e77026e72c","Type":"ContainerStarted","Data":"5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151"}
Oct 31 08:00:13.563672 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:13.563514    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" event={"ID":"79e6087c-f70d-4e91-b0f3-c0e77026e72c","Type":"ContainerStarted","Data":"da406a669d24af22bf285393fbe44c7d55f584ffe47f2c8317b865554abc5471"}
Oct 31 08:00:18.585442 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:18.585407    4913 generic.go:356] "Generic (PLEG): container finished" podID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerID="5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151" exitCode=0
Oct 31 08:00:18.585815 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:18.585481    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" event={"ID":"79e6087c-f70d-4e91-b0f3-c0e77026e72c","Type":"ContainerDied","Data":"5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151"}
Oct 31 08:00:19.399432 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:00:19.399398    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:00:19.482813 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:19.482766    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:00:19.593246 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:19.593212    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" event={"ID":"79e6087c-f70d-4e91-b0f3-c0e77026e72c","Type":"ContainerStarted","Data":"953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932"}
Oct 31 08:00:19.593667 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:19.593276    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" event={"ID":"79e6087c-f70d-4e91-b0f3-c0e77026e72c","Type":"ContainerStarted","Data":"ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06"}
Oct 31 08:00:19.593667 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:19.593495    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:19.613640 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:19.613577    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" podStartSLOduration=1.929262163 podStartE2EDuration="7.613561537s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.684299374s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:00:19.61273974 +0000 UTC m=+1868.302508033" watchObservedRunningTime="2026-10-31 08:00:19.613561537 +0000 UTC m=+1868.303329827"
Oct 31 08:00:20.598319 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:20.598280    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:26.608141 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:26.608110    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:00:29.482481 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:29.482442    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:00:39.482948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:39.482910    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:00:49.399699 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:00:49.399649    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:00:49.482628 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:49.482585    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:00:59.483103 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:00:59.483064    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:01:09.483265 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:09.483225    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:01:13.419413 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:13.419386    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 08:01:18.448102 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:18.448070    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"]
Oct 31 08:01:18.448545 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:18.448320    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" containerID="cri-o://53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6" gracePeriod=30
Oct 31 08:01:18.448545 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:18.448357    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kube-rbac-proxy" containerID="cri-o://dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7" gracePeriod=30
Oct 31 08:01:18.862850 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:18.862768    4913 generic.go:356] "Generic (PLEG): container finished" podID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerID="dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7" exitCode=0
Oct 31 08:01:18.863078 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:18.862842    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" event={"ID":"1db35884-c1ee-43a8-8a06-e9e9f41df5b9","Type":"ContainerDied","Data":"dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7"}
Oct 31 08:01:19.399807 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:01:19.399768    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:01:19.478821 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:19.478780    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Oct 31 08:01:23.415046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.414948    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Oct 31 08:01:23.688345 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.688317    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 08:01:23.826737 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.826702    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-cabundle-cert\") pod \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") "
Oct 31 08:01:23.826911 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.826752    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kserve-provision-location\") pod \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") "
Oct 31 08:01:23.826911 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.826787    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-proxy-tls\") pod \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") "
Oct 31 08:01:23.826911 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.826816    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-rbac-proxy-sar-config\") pod \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") "
Oct 31 08:01:23.826911 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.826846    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-api-access-xbdtc\" (UniqueName: \"kubernetes.io/projected/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-api-access-xbdtc\") pod \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\" (UID: \"1db35884-c1ee-43a8-8a06-e9e9f41df5b9\") "
Oct 31 08:01:23.827159 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.827097    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kserve-provision-location" pod "1db35884-c1ee-43a8-8a06-e9e9f41df5b9" (UID: "1db35884-c1ee-43a8-8a06-e9e9f41df5b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:01:23.827242 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.827167    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-cabundle-cert" pod "1db35884-c1ee-43a8-8a06-e9e9f41df5b9" (UID: "1db35884-c1ee-43a8-8a06-e9e9f41df5b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:01:23.827242 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.827205    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-rbac-proxy-sar-config" pod "1db35884-c1ee-43a8-8a06-e9e9f41df5b9" (UID: "1db35884-c1ee-43a8-8a06-e9e9f41df5b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:01:23.828991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.828963    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-api-access-xbdtc" pod "1db35884-c1ee-43a8-8a06-e9e9f41df5b9" (UID: "1db35884-c1ee-43a8-8a06-e9e9f41df5b9"). InnerVolumeSpecName "kube-api-access-xbdtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:01:23.829066 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.828991    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-proxy-tls" pod "1db35884-c1ee-43a8-8a06-e9e9f41df5b9" (UID: "1db35884-c1ee-43a8-8a06-e9e9f41df5b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:01:23.889638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.889605    4913 generic.go:356] "Generic (PLEG): container finished" podID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerID="53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6" exitCode=0
Oct 31 08:01:23.889796 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.889667    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"
Oct 31 08:01:23.889796 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.889689    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" event={"ID":"1db35884-c1ee-43a8-8a06-e9e9f41df5b9","Type":"ContainerDied","Data":"53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6"}
Oct 31 08:01:23.889796 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.889751    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm" event={"ID":"1db35884-c1ee-43a8-8a06-e9e9f41df5b9","Type":"ContainerDied","Data":"a4f5e17cded05191543557e7c66c2c55b68ff3a1b97815c3bc001cebc91875c2"}
Oct 31 08:01:23.889796 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.889777    4913 scope.go:122] "RemoveContainer" containerID="dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7"
Oct 31 08:01:23.898400 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.898384    4913 scope.go:122] "RemoveContainer" containerID="53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6"
Oct 31 08:01:23.905561 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.905540    4913 scope.go:122] "RemoveContainer" containerID="e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03"
Oct 31 08:01:23.910318 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.910298    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"]
Oct 31 08:01:23.913122 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.913109    4913 scope.go:122] "RemoveContainer" containerID="dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7"
Oct 31 08:01:23.913439 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:01:23.913413    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7\": container with ID starting with dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7 not found: ID does not exist" containerID="dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7"
Oct 31 08:01:23.913522 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.913448    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7"} err="failed to get container status \"dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7\": rpc error: code = NotFound desc = could not find container \"dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7\": container with ID starting with dce93f97e7f1da4ee192254b4a3862472540a84bfe4cd85fc17473c6f7c6ecb7 not found: ID does not exist"
Oct 31 08:01:23.913522 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.913466    4913 scope.go:122] "RemoveContainer" containerID="53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6"
Oct 31 08:01:23.913824 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:01:23.913781    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6\": container with ID starting with 53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6 not found: ID does not exist" containerID="53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6"
Oct 31 08:01:23.913824 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.913821    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6"} err="failed to get container status \"53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6\": rpc error: code = NotFound desc = could not find container \"53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6\": container with ID starting with 53a88b2ee0c03c0511195e9d081a7a1ef7cfcdc6b8c93c33c6b86ac43d4c08a6 not found: ID does not exist"
Oct 31 08:01:23.913824 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.913844    4913 scope.go:122] "RemoveContainer" containerID="e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03"
Oct 31 08:01:23.914428 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:01:23.914248    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03\": container with ID starting with e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03 not found: ID does not exist" containerID="e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03"
Oct 31 08:01:23.914428 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.914280    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03"} err="failed to get container status \"e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03\": rpc error: code = NotFound desc = could not find container \"e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03\": container with ID starting with e11baa9a3584466a69c17cde7cd6df7ad940eeea9076dd922141763802ea6a03 not found: ID does not exist"
Oct 31 08:01:23.915628 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.915609    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-v2-mixed-predictor-7bf65ff98c-gfcxm"]
Oct 31 08:01:23.928319 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.928299    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbdtc\" (UniqueName: \"kubernetes.io/projected/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-api-access-xbdtc\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:01:23.928418 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.928321    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:01:23.928418 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.928332    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:01:23.928418 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.928341    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:01:23.928418 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:23.928350    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db35884-c1ee-43a8-8a06-e9e9f41df5b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:01:25.420437 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:25.420395    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" path="/var/lib/kubelet/pods/1db35884-c1ee-43a8-8a06-e9e9f41df5b9/volumes"
Oct 31 08:01:49.400197 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:01:49.400147    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:01:51.448856 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:51.448820    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:01:59.110399 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:59.110367    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"]
Oct 31 08:01:59.110833 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:59.110616    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="kserve-container" containerID="cri-o://ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06" gracePeriod=30
Oct 31 08:01:59.110833 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:01:59.110697    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="kube-rbac-proxy" containerID="cri-o://953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932" gracePeriod=30
Oct 31 08:02:00.050003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.049972    4913 generic.go:356] "Generic (PLEG): container finished" podID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerID="953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932" exitCode=0
Oct 31 08:02:00.050175 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.050049    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" event={"ID":"79e6087c-f70d-4e91-b0f3-c0e77026e72c","Type":"ContainerDied","Data":"953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932"}
Oct 31 08:02:00.561427 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.561399    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:02:00.637974 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.637892    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-api-access-8fpj5\" (UniqueName: \"kubernetes.io/projected/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-api-access-8fpj5\") pod \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") "
Oct 31 08:02:00.637974 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.637955    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kserve-provision-location\") pod \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") "
Oct 31 08:02:00.638168 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.638004    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-rbac-proxy-sar-config\") pod \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") "
Oct 31 08:02:00.638168 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.638042    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e6087c-f70d-4e91-b0f3-c0e77026e72c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/79e6087c-f70d-4e91-b0f3-c0e77026e72c-proxy-tls\") pod \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") "
Oct 31 08:02:00.638168 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.638079    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-cabundle-cert\") pod \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\" (UID: \"79e6087c-f70d-4e91-b0f3-c0e77026e72c\") "
Oct 31 08:02:00.638532 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.638506    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-rbac-proxy-sar-config" pod "79e6087c-f70d-4e91-b0f3-c0e77026e72c" (UID: "79e6087c-f70d-4e91-b0f3-c0e77026e72c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:02:00.638612 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.638537    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-cabundle-cert" pod "79e6087c-f70d-4e91-b0f3-c0e77026e72c" (UID: "79e6087c-f70d-4e91-b0f3-c0e77026e72c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:02:00.640061 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.640032    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-api-access-8fpj5" pod "79e6087c-f70d-4e91-b0f3-c0e77026e72c" (UID: "79e6087c-f70d-4e91-b0f3-c0e77026e72c"). InnerVolumeSpecName "kube-api-access-8fpj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:02:00.640279 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.640126    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e6087c-f70d-4e91-b0f3-c0e77026e72c-proxy-tls" pod "79e6087c-f70d-4e91-b0f3-c0e77026e72c" (UID: "79e6087c-f70d-4e91-b0f3-c0e77026e72c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:02:00.642163 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.642138    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kserve-provision-location" pod "79e6087c-f70d-4e91-b0f3-c0e77026e72c" (UID: "79e6087c-f70d-4e91-b0f3-c0e77026e72c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:02:00.739402 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.739373    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fpj5\" (UniqueName: \"kubernetes.io/projected/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-api-access-8fpj5\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:00.739402 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.739399    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:00.739402 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.739410    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:00.739639 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.739420    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79e6087c-f70d-4e91-b0f3-c0e77026e72c-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:00.739639 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:00.739430    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79e6087c-f70d-4e91-b0f3-c0e77026e72c-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:02:01.056381 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.056349    4913 generic.go:356] "Generic (PLEG): container finished" podID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerID="ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06" exitCode=0
Oct 31 08:02:01.056596 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.056415    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"
Oct 31 08:02:01.056596 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.056432    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" event={"ID":"79e6087c-f70d-4e91-b0f3-c0e77026e72c","Type":"ContainerDied","Data":"ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06"}
Oct 31 08:02:01.056596 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.056524    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj" event={"ID":"79e6087c-f70d-4e91-b0f3-c0e77026e72c","Type":"ContainerDied","Data":"da406a669d24af22bf285393fbe44c7d55f584ffe47f2c8317b865554abc5471"}
Oct 31 08:02:01.056596 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.056553    4913 scope.go:122] "RemoveContainer" containerID="953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932"
Oct 31 08:02:01.065502 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.065486    4913 scope.go:122] "RemoveContainer" containerID="ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06"
Oct 31 08:02:01.075295 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.075275    4913 scope.go:122] "RemoveContainer" containerID="5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151"
Oct 31 08:02:01.080110 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.080084    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"]
Oct 31 08:02:01.083672 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.083654    4913 scope.go:122] "RemoveContainer" containerID="953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932"
Oct 31 08:02:01.083949 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:02:01.083931    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932\": container with ID starting with 953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932 not found: ID does not exist" containerID="953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932"
Oct 31 08:02:01.084008 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.083959    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932"} err="failed to get container status \"953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932\": rpc error: code = NotFound desc = could not find container \"953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932\": container with ID starting with 953d4636476167357fa6e28f819f9d5b7f0da0ed5ecfce6c4948895ba82d6932 not found: ID does not exist"
Oct 31 08:02:01.084008 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.083981    4913 scope.go:122] "RemoveContainer" containerID="ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06"
Oct 31 08:02:01.084281 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:02:01.084171    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06\": container with ID starting with ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06 not found: ID does not exist" containerID="ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06"
Oct 31 08:02:01.084281 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.084207    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06"} err="failed to get container status \"ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06\": rpc error: code = NotFound desc = could not find container \"ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06\": container with ID starting with ab0edc9f357402ff1425bd387c9f31639f24a23bbdc0f7b3da2f1bef01f64c06 not found: ID does not exist"
Oct 31 08:02:01.084281 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.084219    4913 scope.go:122] "RemoveContainer" containerID="5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151"
Oct 31 08:02:01.084281 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.084236    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-variants-predictor-68877fd588-dc2dj"]
Oct 31 08:02:01.084489 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:02:01.084479    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151\": container with ID starting with 5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151 not found: ID does not exist" containerID="5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151"
Oct 31 08:02:01.084525 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.084495    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151"} err="failed to get container status \"5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151\": rpc error: code = NotFound desc = could not find container \"5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151\": container with ID starting with 5d37651407c7a792e814aabf20398a575e81876f0552e3a2f86621dc236ad151 not found: ID does not exist"
Oct 31 08:02:01.419102 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.419022    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" path="/var/lib/kubelet/pods/79e6087c-f70d-4e91-b0f3-c0e77026e72c/volumes"
Oct 31 08:02:01.649277 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649237    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"]
Oct 31 08:02:01.649780 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649727    4913 cpu_manager.go:414] "containerMap: removing container" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="kube-rbac-proxy"
Oct 31 08:02:01.649780 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649746    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="kube-rbac-proxy"
Oct 31 08:02:01.649780 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649774    4913 cpu_manager.go:414] "containerMap: removing container" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="storage-initializer"
Oct 31 08:02:01.649780 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649781    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="storage-initializer"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649794    4913 cpu_manager.go:414] "containerMap: removing container" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649801    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649817    4913 cpu_manager.go:414] "containerMap: removing container" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="storage-initializer"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649825    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="storage-initializer"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649846    4913 cpu_manager.go:414] "containerMap: removing container" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kube-rbac-proxy"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649854    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kube-rbac-proxy"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649868    4913 cpu_manager.go:414] "containerMap: removing container" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="kserve-container"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649875    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="kserve-container"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649958    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kube-rbac-proxy"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649972    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="kserve-container"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649983    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e6087c-f70d-4e91-b0f3-c0e77026e72c" containerName="kube-rbac-proxy"
Oct 31 08:02:01.650003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.649995    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db35884-c1ee-43a8-8a06-e9e9f41df5b9" containerName="kserve-container"
Oct 31 08:02:01.655067 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.655045    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.663991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.663884    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Oct 31 08:02:01.663991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.663914    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:02:01.663991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.663914    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:02:01.663991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.663923    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:02:01.663991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.663949    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Oct 31 08:02:01.663991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.663951    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:02:01.663991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.663992    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:02:01.666358 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.666335    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"]
Oct 31 08:02:01.748543 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.748505    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.748543 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.748550    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.748794 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.748634    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.748794 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.748671    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.748794 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.748703    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v94vb\" (UniqueName: \"kubernetes.io/projected/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-api-access-v94vb\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.849566 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.849519    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v94vb\" (UniqueName: \"kubernetes.io/projected/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-api-access-v94vb\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.849737 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.849592    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.849737 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.849616    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.849737 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.849639    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.849737 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.849659    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.850092 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.850070    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.850436 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.850411    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.850556 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.850446    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.852302 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.852284    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.857373 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.857356    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v94vb\" (UniqueName: \"kubernetes.io/projected/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-api-access-v94vb\") pod \"isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") " pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:01.973881 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:01.973851    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:02.249238 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:02.249210    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"]
Oct 31 08:02:02.251898 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:02:02.251871    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c2e4ce_26b0_4e52_959a_18ffec9bfe74.slice/crio-f0c40a0366f8590b2bf98d4b26a50441ee8227ee76304c380b5b8741c867b3ed WatchSource:0}: Error finding container f0c40a0366f8590b2bf98d4b26a50441ee8227ee76304c380b5b8741c867b3ed: Status 404 returned error can't find the container with id f0c40a0366f8590b2bf98d4b26a50441ee8227ee76304c380b5b8741c867b3ed
Oct 31 08:02:03.066984 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:03.066947    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" event={"ID":"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74","Type":"ContainerStarted","Data":"3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005"}
Oct 31 08:02:03.067379 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:03.067006    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" event={"ID":"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74","Type":"ContainerStarted","Data":"f0c40a0366f8590b2bf98d4b26a50441ee8227ee76304c380b5b8741c867b3ed"}
Oct 31 08:02:09.093938 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:09.093905    4913 generic.go:356] "Generic (PLEG): container finished" podID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerID="3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005" exitCode=0
Oct 31 08:02:09.094347 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:09.093980    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" event={"ID":"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74","Type":"ContainerDied","Data":"3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005"}
Oct 31 08:02:10.100717 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:10.100681    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" event={"ID":"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74","Type":"ContainerStarted","Data":"e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082"}
Oct 31 08:02:10.101231 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:10.100754    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" event={"ID":"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74","Type":"ContainerStarted","Data":"c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96"}
Oct 31 08:02:10.101231 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:10.100973    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:10.119633 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:10.119584    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" podStartSLOduration=3.19928132 podStartE2EDuration="9.119569713s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.920288393s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:02:10.118138802 +0000 UTC m=+1978.807907095" watchObservedRunningTime="2026-10-31 08:02:10.119569713 +0000 UTC m=+1978.809338004"
Oct 31 08:02:11.105250 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:11.105210    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:11.718291 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:11.718259    4913 scope.go:122] "RemoveContainer" containerID="c20737c719dbaf46386e6816ef65fe53acc7ea5daf171494410f304e4ac1f4a1"
Oct 31 08:02:11.726738 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:11.726713    4913 scope.go:122] "RemoveContainer" containerID="26a1ea955801985b409fea9ee179d290b881c31ca7bc3d4e0083499c31eda771"
Oct 31 08:02:11.735511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:11.735487    4913 scope.go:122] "RemoveContainer" containerID="60abe59b5dbd35d0caf205284a3800914edaff260bae323030f1f64920e6e7b2"
Oct 31 08:02:17.115892 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:02:17.115858    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:02:19.400581 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:02:19.400536    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:02:49.400600 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:02:49.400553    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:03:19.400030 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:03:19.399951    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:03:43.448162 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:43.448125    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:03:49.400154 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:03:49.400118    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:03:54.456412 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:54.456359    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.64:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:03:58.043949 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:58.043909    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"]
Oct 31 08:03:58.044456 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:58.044262    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kserve-container" containerID="cri-o://c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96" gracePeriod=30
Oct 31 08:03:58.044456 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:58.044269    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kube-rbac-proxy" containerID="cri-o://e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082" gracePeriod=30
Oct 31 08:03:58.548263 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:58.548229    4913 generic.go:356] "Generic (PLEG): container finished" podID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerID="e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082" exitCode=0
Oct 31 08:03:58.548482 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:58.548291    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" event={"ID":"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74","Type":"ContainerDied","Data":"e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082"}
Oct 31 08:03:59.491807 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.491781    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:03:59.553979 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.553899    4913 generic.go:356] "Generic (PLEG): container finished" podID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerID="c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96" exitCode=0
Oct 31 08:03:59.553979 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.553961    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"
Oct 31 08:03:59.553979 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.553968    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" event={"ID":"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74","Type":"ContainerDied","Data":"c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96"}
Oct 31 08:03:59.554255 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.554024    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn" event={"ID":"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74","Type":"ContainerDied","Data":"f0c40a0366f8590b2bf98d4b26a50441ee8227ee76304c380b5b8741c867b3ed"}
Oct 31 08:03:59.554255 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.554054    4913 scope.go:122] "RemoveContainer" containerID="e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082"
Oct 31 08:03:59.562082 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.562062    4913 scope.go:122] "RemoveContainer" containerID="c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96"
Oct 31 08:03:59.571523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.571501    4913 scope.go:122] "RemoveContainer" containerID="3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005"
Oct 31 08:03:59.578644 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.578624    4913 scope.go:122] "RemoveContainer" containerID="e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082"
Oct 31 08:03:59.578901 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:03:59.578884    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082\": container with ID starting with e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082 not found: ID does not exist" containerID="e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082"
Oct 31 08:03:59.578948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.578911    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082"} err="failed to get container status \"e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082\": rpc error: code = NotFound desc = could not find container \"e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082\": container with ID starting with e6cc9cb24f5b6533065cdd1944456fc547856fa14fdf2280ad3d226d10eab082 not found: ID does not exist"
Oct 31 08:03:59.578948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.578929    4913 scope.go:122] "RemoveContainer" containerID="c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96"
Oct 31 08:03:59.579145 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:03:59.579126    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96\": container with ID starting with c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96 not found: ID does not exist" containerID="c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96"
Oct 31 08:03:59.579266 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.579152    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96"} err="failed to get container status \"c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96\": rpc error: code = NotFound desc = could not find container \"c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96\": container with ID starting with c938614e6eb59313ec0b91d961bf19e3ae918807fd307a313208d3b190948e96 not found: ID does not exist"
Oct 31 08:03:59.579266 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.579170    4913 scope.go:122] "RemoveContainer" containerID="3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005"
Oct 31 08:03:59.579460 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:03:59.579444    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005\": container with ID starting with 3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005 not found: ID does not exist" containerID="3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005"
Oct 31 08:03:59.579509 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.579468    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005"} err="failed to get container status \"3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005\": rpc error: code = NotFound desc = could not find container \"3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005\": container with ID starting with 3c8619bbc77c8f041bb5d3cfdff1945e847f38ea42e52b40e704358bf091d005 not found: ID does not exist"
Oct 31 08:03:59.606886 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.606861    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-api-access-v94vb\" (UniqueName: \"kubernetes.io/projected/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-api-access-v94vb\") pod \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") "
Oct 31 08:03:59.606993 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.606927    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-proxy-tls\") pod \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") "
Oct 31 08:03:59.606993 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.606959    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kserve-provision-location\") pod \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") "
Oct 31 08:03:59.607089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.607067    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-cabundle-cert\") pod \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") "
Oct 31 08:03:59.607137 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.607125    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-rbac-proxy-sar-config\") pod \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\" (UID: \"d4c2e4ce-26b0-4e52-959a-18ffec9bfe74\") "
Oct 31 08:03:59.607468 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.607436    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-rbac-proxy-sar-config" pod "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" (UID: "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:03:59.607579 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.607478    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-cabundle-cert" pod "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" (UID: "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:03:59.608983 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.608959    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-proxy-tls" pod "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" (UID: "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:03:59.609068 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.609015    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-api-access-v94vb" pod "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" (UID: "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74"). InnerVolumeSpecName "kube-api-access-v94vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:03:59.611838 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.611817    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kserve-provision-location" pod "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" (UID: "d4c2e4ce-26b0-4e52-959a-18ffec9bfe74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:59.708259 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.708217    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:59.708259 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.708256    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:59.708465 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.708274    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:59.708465 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.708287    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:59.708465 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.708299    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v94vb\" (UniqueName: \"kubernetes.io/projected/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74-kube-api-access-v94vb\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:03:59.875477 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.875448    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"]
Oct 31 08:03:59.879302 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:03:59.879275    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-v2-noslash-predictor-57f79f76b9-ql9gn"]
Oct 31 08:04:00.472983 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.472947    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"]
Oct 31 08:04:00.473323 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.473307    4913 cpu_manager.go:414] "containerMap: removing container" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="storage-initializer"
Oct 31 08:04:00.473375 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.473324    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="storage-initializer"
Oct 31 08:04:00.473375 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.473336    4913 cpu_manager.go:414] "containerMap: removing container" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kube-rbac-proxy"
Oct 31 08:04:00.473375 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.473342    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kube-rbac-proxy"
Oct 31 08:04:00.473375 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.473353    4913 cpu_manager.go:414] "containerMap: removing container" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kserve-container"
Oct 31 08:04:00.473375 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.473358    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kserve-container"
Oct 31 08:04:00.473532 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.473424    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kube-rbac-proxy"
Oct 31 08:04:00.473532 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.473432    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" containerName="kserve-container"
Oct 31 08:04:00.478359 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.478338    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.481702 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.481676    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:04:00.482330 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.482297    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Oct 31 08:04:00.482330 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.482327    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:04:00.484686 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.484669    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:04:00.486346 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.486325    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"]
Oct 31 08:04:00.487514 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.487493    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Oct 31 08:04:00.487514 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.487508    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:04:00.487643 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.487581    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:04:00.616291 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.616256    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.616699 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.616301    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgxv\" (UniqueName: \"kubernetes.io/projected/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-api-access-ddgxv\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.616699 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.616411    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.616699 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.616444    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993f2d34-cc57-465d-be6d-4ef2a0b2a973-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.616699 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.616475    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.717527 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.717491    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.717697 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.717533    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993f2d34-cc57-465d-be6d-4ef2a0b2a973-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.717697 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.717658    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.717822 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.717710    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.717822 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.717750    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgxv\" (UniqueName: \"kubernetes.io/projected/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-api-access-ddgxv\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.717931 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.717899    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.718354 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.718329    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.718354 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.718351    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.719953 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.719935    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993f2d34-cc57-465d-be6d-4ef2a0b2a973-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.726985 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.726933    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgxv\" (UniqueName: \"kubernetes.io/projected/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-api-access-ddgxv\") pod \"isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:00.797163 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:00.797130    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:01.067907 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:01.067834    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"]
Oct 31 08:04:01.071764 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:04:01.071729    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993f2d34_cc57_465d_be6d_4ef2a0b2a973.slice/crio-c264e85cb806a23d1c77fcff5e7bb6a804c6a95c2e59fa6a7628341575ac5632 WatchSource:0}: Error finding container c264e85cb806a23d1c77fcff5e7bb6a804c6a95c2e59fa6a7628341575ac5632: Status 404 returned error can't find the container with id c264e85cb806a23d1c77fcff5e7bb6a804c6a95c2e59fa6a7628341575ac5632
Oct 31 08:04:01.421374 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:01.421345    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c2e4ce-26b0-4e52-959a-18ffec9bfe74" path="/var/lib/kubelet/pods/d4c2e4ce-26b0-4e52-959a-18ffec9bfe74/volumes"
Oct 31 08:04:01.564525 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:01.564487    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" event={"ID":"993f2d34-cc57-465d-be6d-4ef2a0b2a973","Type":"ContainerStarted","Data":"3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899"}
Oct 31 08:04:01.564698 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:01.564546    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" event={"ID":"993f2d34-cc57-465d-be6d-4ef2a0b2a973","Type":"ContainerStarted","Data":"c264e85cb806a23d1c77fcff5e7bb6a804c6a95c2e59fa6a7628341575ac5632"}
Oct 31 08:04:06.587171 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:06.587138    4913 generic.go:356] "Generic (PLEG): container finished" podID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerID="3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899" exitCode=0
Oct 31 08:04:06.587606 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:06.587215    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" event={"ID":"993f2d34-cc57-465d-be6d-4ef2a0b2a973","Type":"ContainerDied","Data":"3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899"}
Oct 31 08:04:07.592998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:07.592963    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" event={"ID":"993f2d34-cc57-465d-be6d-4ef2a0b2a973","Type":"ContainerStarted","Data":"c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c"}
Oct 31 08:04:07.593427 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:07.593023    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" event={"ID":"993f2d34-cc57-465d-be6d-4ef2a0b2a973","Type":"ContainerStarted","Data":"09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919"}
Oct 31 08:04:07.593427 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:07.593311    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:07.593508 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:07.593468    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:07.594960 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:07.594927    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 31 08:04:07.613324 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:07.613279    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" podStartSLOduration=2.579271957 podStartE2EDuration="7.613265766s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.033993809s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:04:07.611606542 +0000 UTC m=+2096.301374834" watchObservedRunningTime="2026-10-31 08:04:07.613265766 +0000 UTC m=+2096.303034057"
Oct 31 08:04:08.597595 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:08.597545    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 31 08:04:11.599837 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:11.599809    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:04:11.603300 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:11.603279    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:04:13.602234 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:13.602201    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:13.602802 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:13.602773    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 31 08:04:19.399488 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:04:19.399448    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:04:23.603965 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:23.603931    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:43.246221 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:43.246167    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"]
Oct 31 08:04:43.246716 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:43.246476    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kserve-container" containerID="cri-o://09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919" gracePeriod=30
Oct 31 08:04:43.246978 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:43.246838    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kube-rbac-proxy" containerID="cri-o://c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c" gracePeriod=30
Oct 31 08:04:43.598378 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:43.598275    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused"
Oct 31 08:04:43.603046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:43.603008    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Oct 31 08:04:43.751917 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:43.751874    4913 generic.go:356] "Generic (PLEG): container finished" podID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerID="c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c" exitCode=0
Oct 31 08:04:43.752082 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:43.751940    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" event={"ID":"993f2d34-cc57-465d-be6d-4ef2a0b2a973","Type":"ContainerDied","Data":"c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c"}
Oct 31 08:04:45.000095 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.000069    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:45.093305 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.093228    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-api-access-ddgxv\" (UniqueName: \"kubernetes.io/projected/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-api-access-ddgxv\") pod \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") "
Oct 31 08:04:45.093470 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.093308    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993f2d34-cc57-465d-be6d-4ef2a0b2a973-proxy-tls\" (UniqueName: \"kubernetes.io/secret/993f2d34-cc57-465d-be6d-4ef2a0b2a973-proxy-tls\") pod \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") "
Oct 31 08:04:45.093470 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.093329    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-rbac-proxy-sar-config\") pod \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") "
Oct 31 08:04:45.093470 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.093347    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kserve-provision-location\") pod \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") "
Oct 31 08:04:45.093470 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.093381    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-cabundle-cert\") pod \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\" (UID: \"993f2d34-cc57-465d-be6d-4ef2a0b2a973\") "
Oct 31 08:04:45.093833 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.093800    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-rbac-proxy-sar-config" pod "993f2d34-cc57-465d-be6d-4ef2a0b2a973" (UID: "993f2d34-cc57-465d-be6d-4ef2a0b2a973"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:04:45.093896 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.093807    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-cabundle-cert" pod "993f2d34-cc57-465d-be6d-4ef2a0b2a973" (UID: "993f2d34-cc57-465d-be6d-4ef2a0b2a973"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:04:45.094473 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.094445    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kserve-provision-location" pod "993f2d34-cc57-465d-be6d-4ef2a0b2a973" (UID: "993f2d34-cc57-465d-be6d-4ef2a0b2a973"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:04:45.095456 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.095432    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-api-access-ddgxv" pod "993f2d34-cc57-465d-be6d-4ef2a0b2a973" (UID: "993f2d34-cc57-465d-be6d-4ef2a0b2a973"). InnerVolumeSpecName "kube-api-access-ddgxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:04:45.095563 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.095510    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993f2d34-cc57-465d-be6d-4ef2a0b2a973-proxy-tls" pod "993f2d34-cc57-465d-be6d-4ef2a0b2a973" (UID: "993f2d34-cc57-465d-be6d-4ef2a0b2a973"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:04:45.194750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.194696    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/993f2d34-cc57-465d-be6d-4ef2a0b2a973-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:04:45.194750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.194745    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:04:45.194750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.194758    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:04:45.194750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.194767    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/993f2d34-cc57-465d-be6d-4ef2a0b2a973-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:04:45.195025 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.194778    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddgxv\" (UniqueName: \"kubernetes.io/projected/993f2d34-cc57-465d-be6d-4ef2a0b2a973-kube-api-access-ddgxv\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:04:45.768146 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.768064    4913 generic.go:356] "Generic (PLEG): container finished" podID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerID="09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919" exitCode=0
Oct 31 08:04:45.768146 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.768115    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" event={"ID":"993f2d34-cc57-465d-be6d-4ef2a0b2a973","Type":"ContainerDied","Data":"09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919"}
Oct 31 08:04:45.768146 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.768130    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"
Oct 31 08:04:45.768395 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.768164    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h" event={"ID":"993f2d34-cc57-465d-be6d-4ef2a0b2a973","Type":"ContainerDied","Data":"c264e85cb806a23d1c77fcff5e7bb6a804c6a95c2e59fa6a7628341575ac5632"}
Oct 31 08:04:45.768395 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.768208    4913 scope.go:122] "RemoveContainer" containerID="c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c"
Oct 31 08:04:45.776248 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.776229    4913 scope.go:122] "RemoveContainer" containerID="09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919"
Oct 31 08:04:45.783370 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.783352    4913 scope.go:122] "RemoveContainer" containerID="3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899"
Oct 31 08:04:45.785675 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.785657    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"]
Oct 31 08:04:45.789125 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.789103    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-predictor-85fdb8fc88-dwd5h"]
Oct 31 08:04:45.791167 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.791151    4913 scope.go:122] "RemoveContainer" containerID="c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c"
Oct 31 08:04:45.791435 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:04:45.791416    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c\": container with ID starting with c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c not found: ID does not exist" containerID="c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c"
Oct 31 08:04:45.791516 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.791447    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c"} err="failed to get container status \"c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c\": rpc error: code = NotFound desc = could not find container \"c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c\": container with ID starting with c9d6d93630a44265a657385ac521188063e956841f0b69eef73efdc51996d53c not found: ID does not exist"
Oct 31 08:04:45.791516 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.791472    4913 scope.go:122] "RemoveContainer" containerID="09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919"
Oct 31 08:04:45.791773 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:04:45.791754    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919\": container with ID starting with 09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919 not found: ID does not exist" containerID="09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919"
Oct 31 08:04:45.791816 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.791781    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919"} err="failed to get container status \"09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919\": rpc error: code = NotFound desc = could not find container \"09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919\": container with ID starting with 09ca57fd52aa888c30874c732850b4691a40560d4a14d6ffa19d3db486ee2919 not found: ID does not exist"
Oct 31 08:04:45.791816 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.791797    4913 scope.go:122] "RemoveContainer" containerID="3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899"
Oct 31 08:04:45.792048 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:04:45.792029    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899\": container with ID starting with 3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899 not found: ID does not exist" containerID="3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899"
Oct 31 08:04:45.792099 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:45.792053    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899"} err="failed to get container status \"3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899\": rpc error: code = NotFound desc = could not find container \"3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899\": container with ID starting with 3e4898d768967d72e76befc90688e06bf28f5eeb0efc633fc9239081cb176899 not found: ID does not exist"
Oct 31 08:04:47.418524 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.418492    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" path="/var/lib/kubelet/pods/993f2d34-cc57-465d-be6d-4ef2a0b2a973/volumes"
Oct 31 08:04:47.733581 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733534    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"]
Oct 31 08:04:47.733874 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733863    4913 cpu_manager.go:414] "containerMap: removing container" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kube-rbac-proxy"
Oct 31 08:04:47.733923 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733875    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kube-rbac-proxy"
Oct 31 08:04:47.733923 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733887    4913 cpu_manager.go:414] "containerMap: removing container" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kserve-container"
Oct 31 08:04:47.733923 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733892    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kserve-container"
Oct 31 08:04:47.733923 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733905    4913 cpu_manager.go:414] "containerMap: removing container" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="storage-initializer"
Oct 31 08:04:47.733923 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733911    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="storage-initializer"
Oct 31 08:04:47.734080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733966    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kube-rbac-proxy"
Oct 31 08:04:47.734080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.733975    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="993f2d34-cc57-465d-be6d-4ef2a0b2a973" containerName="kserve-container"
Oct 31 08:04:47.738891 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.738866    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.742413 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.742392    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Oct 31 08:04:47.746286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.745732    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:04:47.746842 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.746820    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Oct 31 08:04:47.746996 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.746857    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:04:47.747080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.746889    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:04:47.748526 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.748504    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"]
Oct 31 08:04:47.815872 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.815832    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gztgj\" (UniqueName: \"kubernetes.io/projected/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-api-access-gztgj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.816046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.815893    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.816046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.815983    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.816046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.816023    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.816163 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.816146    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db0ee03-9f4c-497f-87b8-d215f232a2b3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.842620 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.842589    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:04:47.842620 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.842591    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:04:47.917125 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.917087    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db0ee03-9f4c-497f-87b8-d215f232a2b3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.917298 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.917138    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gztgj\" (UniqueName: \"kubernetes.io/projected/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-api-access-gztgj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.917298 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.917192    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.917298 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.917221    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.917298 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.917246    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.917787 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.917764    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.917952 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.917934    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.918070 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.918045    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.919597 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.919577    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db0ee03-9f4c-497f-87b8-d215f232a2b3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:47.924222 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:47.924178    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gztgj\" (UniqueName: \"kubernetes.io/projected/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-api-access-gztgj\") pod \"isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") " pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:48.056905 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:48.056809    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:04:48.340845 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:48.340766    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"]
Oct 31 08:04:48.343835 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:04:48.343806    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db0ee03_9f4c_497f_87b8_d215f232a2b3.slice/crio-4d5947b67251274e6b3a3aa18ae4a8e9c1d9e8f3c1eb8dc9c2d42e2e32ae1656 WatchSource:0}: Error finding container 4d5947b67251274e6b3a3aa18ae4a8e9c1d9e8f3c1eb8dc9c2d42e2e32ae1656: Status 404 returned error can't find the container with id 4d5947b67251274e6b3a3aa18ae4a8e9c1d9e8f3c1eb8dc9c2d42e2e32ae1656
Oct 31 08:04:48.782660 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:48.782620    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" event={"ID":"1db0ee03-9f4c-497f-87b8-d215f232a2b3","Type":"ContainerStarted","Data":"85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43"}
Oct 31 08:04:48.783127 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:48.782691    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" event={"ID":"1db0ee03-9f4c-497f-87b8-d215f232a2b3","Type":"ContainerStarted","Data":"4d5947b67251274e6b3a3aa18ae4a8e9c1d9e8f3c1eb8dc9c2d42e2e32ae1656"}
Oct 31 08:04:49.399717 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:04:49.399682    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:04:59.828062 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:59.828027    4913 generic.go:356] "Generic (PLEG): container finished" podID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerID="85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43" exitCode=0
Oct 31 08:04:59.828506 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:04:59.828097    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" event={"ID":"1db0ee03-9f4c-497f-87b8-d215f232a2b3","Type":"ContainerDied","Data":"85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43"}
Oct 31 08:05:00.835131 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:00.835097    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" event={"ID":"1db0ee03-9f4c-497f-87b8-d215f232a2b3","Type":"ContainerStarted","Data":"42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b"}
Oct 31 08:05:00.835542 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:00.835159    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" event={"ID":"1db0ee03-9f4c-497f-87b8-d215f232a2b3","Type":"ContainerStarted","Data":"51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02"}
Oct 31 08:05:00.835542 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:00.835356    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:05:00.854249 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:00.854200    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" podStartSLOduration=3.252687191 podStartE2EDuration="13.85417277s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.601485579s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:05:00.853177765 +0000 UTC m=+2149.542946058" watchObservedRunningTime="2026-10-31 08:05:00.85417277 +0000 UTC m=+2149.543941055"
Oct 31 08:05:01.840316 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:01.840283    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:05:01.841916 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:01.841885    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 31 08:05:02.844596 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:02.844558    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 31 08:05:07.850360 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:07.850319    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:05:07.850934 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:07.850908    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Oct 31 08:05:17.852343 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:17.852309    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:05:19.400342 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:05:19.400307    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:05:33.361158 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:33.361120    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"]
Oct 31 08:05:33.361726 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:33.361395    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kserve-container" containerID="cri-o://51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02" gracePeriod=30
Oct 31 08:05:33.361726 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:33.361417    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kube-rbac-proxy" containerID="cri-o://42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b" gracePeriod=30
Oct 31 08:05:33.980523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:33.980487    4913 generic.go:356] "Generic (PLEG): container finished" podID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerID="42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b" exitCode=0
Oct 31 08:05:33.980696 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:33.980530    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" event={"ID":"1db0ee03-9f4c-497f-87b8-d215f232a2b3","Type":"ContainerDied","Data":"42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b"}
Oct 31 08:05:34.809605 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.809579    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:05:34.920248 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.920153    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-api-access-gztgj\" (UniqueName: \"kubernetes.io/projected/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-api-access-gztgj\") pod \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") "
Oct 31 08:05:34.920395 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.920276    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-rbac-proxy-sar-config\") pod \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") "
Oct 31 08:05:34.920395 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.920315    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kserve-provision-location\") pod \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") "
Oct 31 08:05:34.920395 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.920373    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db0ee03-9f4c-497f-87b8-d215f232a2b3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db0ee03-9f4c-497f-87b8-d215f232a2b3-proxy-tls\") pod \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") "
Oct 31 08:05:34.920524 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.920418    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-cabundle-cert\") pod \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\" (UID: \"1db0ee03-9f4c-497f-87b8-d215f232a2b3\") "
Oct 31 08:05:34.920796 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.920763    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-rbac-proxy-sar-config" pod "1db0ee03-9f4c-497f-87b8-d215f232a2b3" (UID: "1db0ee03-9f4c-497f-87b8-d215f232a2b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:05:34.920897 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.920841    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-cabundle-cert" pod "1db0ee03-9f4c-497f-87b8-d215f232a2b3" (UID: "1db0ee03-9f4c-497f-87b8-d215f232a2b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:05:34.921290 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.921261    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kserve-provision-location" pod "1db0ee03-9f4c-497f-87b8-d215f232a2b3" (UID: "1db0ee03-9f4c-497f-87b8-d215f232a2b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:05:34.922462 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.922438    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db0ee03-9f4c-497f-87b8-d215f232a2b3-proxy-tls" pod "1db0ee03-9f4c-497f-87b8-d215f232a2b3" (UID: "1db0ee03-9f4c-497f-87b8-d215f232a2b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:05:34.922514 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.922487    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-api-access-gztgj" pod "1db0ee03-9f4c-497f-87b8-d215f232a2b3" (UID: "1db0ee03-9f4c-497f-87b8-d215f232a2b3"). InnerVolumeSpecName "kube-api-access-gztgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:05:34.986615 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.986582    4913 generic.go:356] "Generic (PLEG): container finished" podID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerID="51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02" exitCode=0
Oct 31 08:05:34.986764 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.986623    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" event={"ID":"1db0ee03-9f4c-497f-87b8-d215f232a2b3","Type":"ContainerDied","Data":"51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02"}
Oct 31 08:05:34.986764 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.986643    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"
Oct 31 08:05:34.986764 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.986701    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr" event={"ID":"1db0ee03-9f4c-497f-87b8-d215f232a2b3","Type":"ContainerDied","Data":"4d5947b67251274e6b3a3aa18ae4a8e9c1d9e8f3c1eb8dc9c2d42e2e32ae1656"}
Oct 31 08:05:34.986764 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.986727    4913 scope.go:122] "RemoveContainer" containerID="42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b"
Oct 31 08:05:34.995135 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:34.995109    4913 scope.go:122] "RemoveContainer" containerID="51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02"
Oct 31 08:05:35.003125 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.003103    4913 scope.go:122] "RemoveContainer" containerID="85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43"
Oct 31 08:05:35.008337 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.008317    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"]
Oct 31 08:05:35.011153 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.011128    4913 scope.go:122] "RemoveContainer" containerID="42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b"
Oct 31 08:05:35.011814 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:05:35.011788    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b\": container with ID starting with 42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b not found: ID does not exist" containerID="42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b"
Oct 31 08:05:35.011919 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.011821    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b"} err="failed to get container status \"42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b\": rpc error: code = NotFound desc = could not find container \"42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b\": container with ID starting with 42e31934b04a5ae0170c989b2b924f8403087fa1087e088af6dcff892211524b not found: ID does not exist"
Oct 31 08:05:35.011919 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.011846    4913 scope.go:122] "RemoveContainer" containerID="51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02"
Oct 31 08:05:35.012125 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:05:35.012102    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02\": container with ID starting with 51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02 not found: ID does not exist" containerID="51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02"
Oct 31 08:05:35.012207 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.012131    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02"} err="failed to get container status \"51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02\": rpc error: code = NotFound desc = could not find container \"51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02\": container with ID starting with 51a051718547c99858b5a12cccf0b5a225f4e121a53be58f03b0f591ffc0ba02 not found: ID does not exist"
Oct 31 08:05:35.012207 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.012146    4913 scope.go:122] "RemoveContainer" containerID="85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43"
Oct 31 08:05:35.012455 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:05:35.012435    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43\": container with ID starting with 85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43 not found: ID does not exist" containerID="85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43"
Oct 31 08:05:35.012529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.012459    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43"} err="failed to get container status \"85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43\": rpc error: code = NotFound desc = could not find container \"85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43\": container with ID starting with 85d2e175239fb6978181c1b9b3e9f53211225bca067f62cefc2c4b70d4e94c43 not found: ID does not exist"
Oct 31 08:05:35.012529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.012478    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-autogluon-ts-v1-noslash-predictor-7c79dc8697-bn7mr"]
Oct 31 08:05:35.021837 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.021821    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gztgj\" (UniqueName: \"kubernetes.io/projected/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-api-access-gztgj\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:05:35.021893 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.021840    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:05:35.021893 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.021850    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1db0ee03-9f4c-497f-87b8-d215f232a2b3-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:05:35.021893 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.021860    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1db0ee03-9f4c-497f-87b8-d215f232a2b3-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:05:35.021893 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.021870    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1db0ee03-9f4c-497f-87b8-d215f232a2b3-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:05:35.419205 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:35.419155    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" path="/var/lib/kubelet/pods/1db0ee03-9f4c-497f-87b8-d215f232a2b3/volumes"
Oct 31 08:05:36.806066 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806035    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"]
Oct 31 08:05:36.806511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806378    4913 cpu_manager.go:414] "containerMap: removing container" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="storage-initializer"
Oct 31 08:05:36.806511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806392    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="storage-initializer"
Oct 31 08:05:36.806511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806407    4913 cpu_manager.go:414] "containerMap: removing container" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kserve-container"
Oct 31 08:05:36.806511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806413    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kserve-container"
Oct 31 08:05:36.806511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806425    4913 cpu_manager.go:414] "containerMap: removing container" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kube-rbac-proxy"
Oct 31 08:05:36.806511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806430    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kube-rbac-proxy"
Oct 31 08:05:36.806511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806487    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kserve-container"
Oct 31 08:05:36.806511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.806495    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db0ee03-9f4c-497f-87b8-d215f232a2b3" containerName="kube-rbac-proxy"
Oct 31 08:05:36.809490 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.809470    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:36.814097 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.814061    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Oct 31 08:05:36.814961 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.814942    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:05:36.814961 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.814942    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:05:36.815317 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.815300    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 31 08:05:36.815415 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.815353    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:05:36.815415 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.815398    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:05:36.815415 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.815408    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:05:36.819408 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.819385    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"]
Oct 31 08:05:36.852767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.852736    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"]
Oct 31 08:05:36.856491 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.856467    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:36.860795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.860773    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 31 08:05:36.865822 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.865794    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"]
Oct 31 08:05:36.938686 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.938653    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482vq\" (UniqueName: \"kubernetes.io/projected/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-api-access-482vq\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:36.938868 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.938707    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-cabundle-cert\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:36.938868 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.938733    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:36.938944 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.938872    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:36.938944 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.938910    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5wx\" (UniqueName: \"kubernetes.io/projected/c0799a12-056d-4863-8545-cf7f73513d3d-kube-api-access-jt5wx\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:36.938944 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.938934    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0799a12-056d-4863-8545-cf7f73513d3d-kserve-provision-location\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:36.939063 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.938967    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-proxy-tls\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:36.939063 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.939027    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:36.939063 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.939059    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0799a12-056d-4863-8545-cf7f73513d3d-proxy-tls\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:36.939286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:36.939211    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.040288 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040249    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-proxy-tls\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.040288 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040291    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.040519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040310    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0799a12-056d-4863-8545-cf7f73513d3d-proxy-tls\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.040519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040336    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.040519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040356    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482vq\" (UniqueName: \"kubernetes.io/projected/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-api-access-482vq\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.040519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040388    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-cabundle-cert\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.040519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040407    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.040519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040438    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.040519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040477    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5wx\" (UniqueName: \"kubernetes.io/projected/c0799a12-056d-4863-8545-cf7f73513d3d-kube-api-access-jt5wx\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.040900 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040530    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0799a12-056d-4863-8545-cf7f73513d3d-kserve-provision-location\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.040900 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040765    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.040900 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.040866    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0799a12-056d-4863-8545-cf7f73513d3d-kserve-provision-location\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.041260 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.041236    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-cabundle-cert\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.041429 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.041395    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.041557 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.041256    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.041557 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.041286    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.043471 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.043448    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-proxy-tls\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.043552 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.043448    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0799a12-056d-4863-8545-cf7f73513d3d-proxy-tls\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.047912 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.047891    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482vq\" (UniqueName: \"kubernetes.io/projected/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-api-access-482vq\") pod \"isvc-canary-create-v2-predictor-694d996ccf-f4ncw\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") " pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.047993 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.047960    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5wx\" (UniqueName: \"kubernetes.io/projected/c0799a12-056d-4863-8545-cf7f73513d3d-kube-api-access-jt5wx\") pod \"isvc-canary-create-predictor-55d965bd48-vs859\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") " pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.129756 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.129675    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:37.175704 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.175665    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:37.404700 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.404625    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"]
Oct 31 08:05:37.407873 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:05:37.407845    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c2c3c4_df8e_41cc_8eaf_7f6b70834902.slice/crio-802de0e00deb8e5c9529cae004ea05576a8ef6280f89c061f1addfa53e7397d8 WatchSource:0}: Error finding container 802de0e00deb8e5c9529cae004ea05576a8ef6280f89c061f1addfa53e7397d8: Status 404 returned error can't find the container with id 802de0e00deb8e5c9529cae004ea05576a8ef6280f89c061f1addfa53e7397d8
Oct 31 08:05:37.454205 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:37.454164    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"]
Oct 31 08:05:37.463210 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:05:37.463167    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0799a12_056d_4863_8545_cf7f73513d3d.slice/crio-74eb41629278fe4b1f2d531e0f9e197e4b606636062160d1a5ff07403e1ce9a5 WatchSource:0}: Error finding container 74eb41629278fe4b1f2d531e0f9e197e4b606636062160d1a5ff07403e1ce9a5: Status 404 returned error can't find the container with id 74eb41629278fe4b1f2d531e0f9e197e4b606636062160d1a5ff07403e1ce9a5
Oct 31 08:05:38.001434 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:38.001392    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" event={"ID":"c0799a12-056d-4863-8545-cf7f73513d3d","Type":"ContainerStarted","Data":"4afccffb62c02d3d80e1b0a3503be3ad93d3f2783a2d26e1979d78e3fe5ee165"}
Oct 31 08:05:38.001926 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:38.001460    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" event={"ID":"c0799a12-056d-4863-8545-cf7f73513d3d","Type":"ContainerStarted","Data":"74eb41629278fe4b1f2d531e0f9e197e4b606636062160d1a5ff07403e1ce9a5"}
Oct 31 08:05:38.003069 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:38.003045    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" event={"ID":"79c2c3c4-df8e-41cc-8eaf-7f6b70834902","Type":"ContainerStarted","Data":"26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011"}
Oct 31 08:05:38.003214 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:38.003103    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" event={"ID":"79c2c3c4-df8e-41cc-8eaf-7f6b70834902","Type":"ContainerStarted","Data":"802de0e00deb8e5c9529cae004ea05576a8ef6280f89c061f1addfa53e7397d8"}
Oct 31 08:05:42.021459 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:42.021422    4913 generic.go:356] "Generic (PLEG): container finished" podID="c0799a12-056d-4863-8545-cf7f73513d3d" containerID="4afccffb62c02d3d80e1b0a3503be3ad93d3f2783a2d26e1979d78e3fe5ee165" exitCode=0
Oct 31 08:05:42.021942 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:42.021497    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" event={"ID":"c0799a12-056d-4863-8545-cf7f73513d3d","Type":"ContainerDied","Data":"4afccffb62c02d3d80e1b0a3503be3ad93d3f2783a2d26e1979d78e3fe5ee165"}
Oct 31 08:05:42.023080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:42.023060    4913 generic.go:356] "Generic (PLEG): container finished" podID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerID="26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011" exitCode=0
Oct 31 08:05:42.023177 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:42.023119    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" event={"ID":"79c2c3c4-df8e-41cc-8eaf-7f6b70834902","Type":"ContainerDied","Data":"26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011"}
Oct 31 08:05:43.030569 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.030537    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" event={"ID":"c0799a12-056d-4863-8545-cf7f73513d3d","Type":"ContainerStarted","Data":"c38bf44231e895ef241a27cae272f0df474f939126479b002126b3a06d04d8bb"}
Oct 31 08:05:43.031022 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.030613    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" event={"ID":"c0799a12-056d-4863-8545-cf7f73513d3d","Type":"ContainerStarted","Data":"79d927a5320d5467de7a90ac10f8e72104b2df3e847953ec4bdeceb835b27111"}
Oct 31 08:05:43.031022 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.030889    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:43.031022 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.030956    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:43.032441 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.032414    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:05:43.032824 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.032805    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" event={"ID":"79c2c3c4-df8e-41cc-8eaf-7f6b70834902","Type":"ContainerStarted","Data":"22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66"}
Oct 31 08:05:43.032877 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.032851    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" event={"ID":"79c2c3c4-df8e-41cc-8eaf-7f6b70834902","Type":"ContainerStarted","Data":"5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435"}
Oct 31 08:05:43.033036 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.033021    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:43.051215 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.051141    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podStartSLOduration=3.040077566 podStartE2EDuration="7.051128536s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.01105097s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:05:43.050344327 +0000 UTC m=+2191.740112640" watchObservedRunningTime="2026-10-31 08:05:43.051128536 +0000 UTC m=+2191.740896827"
Oct 31 08:05:43.068251 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:43.068207    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podStartSLOduration=2.9561310130000003 podStartE2EDuration="7.068176742s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.112045729s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:05:43.066820821 +0000 UTC m=+2191.756589113" watchObservedRunningTime="2026-10-31 08:05:43.068176742 +0000 UTC m=+2191.757945027"
Oct 31 08:05:44.036793 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:44.036754    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:05:44.037283 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:44.036805    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:44.037978 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:44.037953    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:05:45.040447 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:45.040407    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:05:49.041384 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:49.041350    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:05:49.041871 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:49.041845    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:05:49.401037 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:05:49.400948    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:05:50.045425 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:50.045396    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:05:50.045900 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:50.045875    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:05:59.042719 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:05:59.042625    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:06:00.045879 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:00.045842    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:06:09.042131 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:09.042094    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:06:10.045848 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:10.045807    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:06:19.041792 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:19.041753    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:06:19.400434 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:06:19.400352    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:06:20.046357 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:20.046316    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:06:29.041886 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:29.041840    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:06:30.046449 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:30.046411    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:06:39.042042 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:39.042002    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:06:40.046554 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:40.046514    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:06:49.042372 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:49.042332    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:06:49.399986 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:06:49.399903    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:06:50.046866 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:50.046828    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:06:55.414629 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:55.414576    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Oct 31 08:06:58.414173 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:06:58.414121    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Oct 31 08:07:05.419859 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:05.419832    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:07:05.555670 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:05.555629    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"]
Oct 31 08:07:05.555937 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:05.555908    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container" containerID="cri-o://79d927a5320d5467de7a90ac10f8e72104b2df3e847953ec4bdeceb835b27111" gracePeriod=30
Oct 31 08:07:05.556054 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:05.555945    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kube-rbac-proxy" containerID="cri-o://c38bf44231e895ef241a27cae272f0df474f939126479b002126b3a06d04d8bb" gracePeriod=30
Oct 31 08:07:05.556785 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:05.556741    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:07:06.376270 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:06.376235    4913 generic.go:356] "Generic (PLEG): container finished" podID="c0799a12-056d-4863-8545-cf7f73513d3d" containerID="c38bf44231e895ef241a27cae272f0df474f939126479b002126b3a06d04d8bb" exitCode=0
Oct 31 08:07:06.376444 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:06.376304    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" event={"ID":"c0799a12-056d-4863-8545-cf7f73513d3d","Type":"ContainerDied","Data":"c38bf44231e895ef241a27cae272f0df474f939126479b002126b3a06d04d8bb"}
Oct 31 08:07:09.037028 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:09.036984    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.68:8643/healthz\": dial tcp 10.132.0.68:8643: connect: connection refused"
Oct 31 08:07:10.395291 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.395242    4913 generic.go:356] "Generic (PLEG): container finished" podID="c0799a12-056d-4863-8545-cf7f73513d3d" containerID="79d927a5320d5467de7a90ac10f8e72104b2df3e847953ec4bdeceb835b27111" exitCode=0
Oct 31 08:07:10.395756 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.395319    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" event={"ID":"c0799a12-056d-4863-8545-cf7f73513d3d","Type":"ContainerDied","Data":"79d927a5320d5467de7a90ac10f8e72104b2df3e847953ec4bdeceb835b27111"}
Oct 31 08:07:10.501741 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.501716    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:07:10.611769 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.611683    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0799a12-056d-4863-8545-cf7f73513d3d-kube-api-access-jt5wx\" (UniqueName: \"kubernetes.io/projected/c0799a12-056d-4863-8545-cf7f73513d3d-kube-api-access-jt5wx\") pod \"c0799a12-056d-4863-8545-cf7f73513d3d\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") "
Oct 31 08:07:10.611769 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.611731    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0799a12-056d-4863-8545-cf7f73513d3d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0799a12-056d-4863-8545-cf7f73513d3d-kserve-provision-location\") pod \"c0799a12-056d-4863-8545-cf7f73513d3d\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") "
Oct 31 08:07:10.611769 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.611763    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0799a12-056d-4863-8545-cf7f73513d3d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0799a12-056d-4863-8545-cf7f73513d3d-proxy-tls\") pod \"c0799a12-056d-4863-8545-cf7f73513d3d\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") "
Oct 31 08:07:10.612097 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.611788    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-kube-rbac-proxy-sar-config\") pod \"c0799a12-056d-4863-8545-cf7f73513d3d\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") "
Oct 31 08:07:10.612097 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.611812    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-cabundle-cert\") pod \"c0799a12-056d-4863-8545-cf7f73513d3d\" (UID: \"c0799a12-056d-4863-8545-cf7f73513d3d\") "
Oct 31 08:07:10.612233 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.612159    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0799a12-056d-4863-8545-cf7f73513d3d-kserve-provision-location" pod "c0799a12-056d-4863-8545-cf7f73513d3d" (UID: "c0799a12-056d-4863-8545-cf7f73513d3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:10.612281 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.612230    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-kube-rbac-proxy-sar-config" pod "c0799a12-056d-4863-8545-cf7f73513d3d" (UID: "c0799a12-056d-4863-8545-cf7f73513d3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:10.612281 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.612262    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-cabundle-cert" pod "c0799a12-056d-4863-8545-cf7f73513d3d" (UID: "c0799a12-056d-4863-8545-cf7f73513d3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:10.613936 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.613911    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0799a12-056d-4863-8545-cf7f73513d3d-kube-api-access-jt5wx" pod "c0799a12-056d-4863-8545-cf7f73513d3d" (UID: "c0799a12-056d-4863-8545-cf7f73513d3d"). InnerVolumeSpecName "kube-api-access-jt5wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:07:10.614064 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.613947    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0799a12-056d-4863-8545-cf7f73513d3d-proxy-tls" pod "c0799a12-056d-4863-8545-cf7f73513d3d" (UID: "c0799a12-056d-4863-8545-cf7f73513d3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:07:10.712351 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.712318    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5wx\" (UniqueName: \"kubernetes.io/projected/c0799a12-056d-4863-8545-cf7f73513d3d-kube-api-access-jt5wx\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:10.712351 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.712344    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0799a12-056d-4863-8545-cf7f73513d3d-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:10.712351 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.712354    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0799a12-056d-4863-8545-cf7f73513d3d-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:10.712581 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.712363    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:10.712581 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:10.712372    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0799a12-056d-4863-8545-cf7f73513d3d-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:11.400998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:11.400968    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"
Oct 31 08:07:11.401628 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:11.400967    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859" event={"ID":"c0799a12-056d-4863-8545-cf7f73513d3d","Type":"ContainerDied","Data":"74eb41629278fe4b1f2d531e0f9e197e4b606636062160d1a5ff07403e1ce9a5"}
Oct 31 08:07:11.401628 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:11.401104    4913 scope.go:122] "RemoveContainer" containerID="c38bf44231e895ef241a27cae272f0df474f939126479b002126b3a06d04d8bb"
Oct 31 08:07:11.409762 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:11.409741    4913 scope.go:122] "RemoveContainer" containerID="79d927a5320d5467de7a90ac10f8e72104b2df3e847953ec4bdeceb835b27111"
Oct 31 08:07:11.419019 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:11.419000    4913 scope.go:122] "RemoveContainer" containerID="4afccffb62c02d3d80e1b0a3503be3ad93d3f2783a2d26e1979d78e3fe5ee165"
Oct 31 08:07:11.423379 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:11.423358    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"]
Oct 31 08:07:11.425416 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:11.425391    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-predictor-55d965bd48-vs859"]
Oct 31 08:07:13.419077 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:13.419042    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" path="/var/lib/kubelet/pods/c0799a12-056d-4863-8545-cf7f73513d3d/volumes"
Oct 31 08:07:19.400349 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:07:19.400310    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:07:37.108685 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:37.108483    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"]
Oct 31 08:07:37.109161 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:37.108777    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container" containerID="cri-o://5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435" gracePeriod=30
Oct 31 08:07:37.109161 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:37.108829    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kube-rbac-proxy" containerID="cri-o://22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66" gracePeriod=30
Oct 31 08:07:37.502890 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:37.502854    4913 generic.go:356] "Generic (PLEG): container finished" podID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerID="22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66" exitCode=0
Oct 31 08:07:37.503072 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:37.502902    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" event={"ID":"79c2c3c4-df8e-41cc-8eaf-7f6b70834902","Type":"ContainerDied","Data":"22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66"}
Oct 31 08:07:40.040898 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:40.040858    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Oct 31 08:07:42.156249 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.156225    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:07:42.245910 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.245839    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-cabundle-cert\") pod \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") "
Oct 31 08:07:42.245910 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.245883    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-api-access-482vq\" (UniqueName: \"kubernetes.io/projected/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-api-access-482vq\") pod \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") "
Oct 31 08:07:42.246168 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.245917    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kserve-provision-location\") pod \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") "
Oct 31 08:07:42.246168 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.246025    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-rbac-proxy-sar-config\") pod \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") "
Oct 31 08:07:42.246168 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.246053    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-proxy-tls\" (UniqueName: \"kubernetes.io/secret/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-proxy-tls\") pod \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\" (UID: \"79c2c3c4-df8e-41cc-8eaf-7f6b70834902\") "
Oct 31 08:07:42.246359 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.246299    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kserve-provision-location" pod "79c2c3c4-df8e-41cc-8eaf-7f6b70834902" (UID: "79c2c3c4-df8e-41cc-8eaf-7f6b70834902"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:42.246418 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.246362    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-cabundle-cert" pod "79c2c3c4-df8e-41cc-8eaf-7f6b70834902" (UID: "79c2c3c4-df8e-41cc-8eaf-7f6b70834902"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:42.246418 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.246389    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-rbac-proxy-sar-config" pod "79c2c3c4-df8e-41cc-8eaf-7f6b70834902" (UID: "79c2c3c4-df8e-41cc-8eaf-7f6b70834902"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:42.248028 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.248005    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-api-access-482vq" pod "79c2c3c4-df8e-41cc-8eaf-7f6b70834902" (UID: "79c2c3c4-df8e-41cc-8eaf-7f6b70834902"). InnerVolumeSpecName "kube-api-access-482vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:07:42.248122 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.248083    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-proxy-tls" pod "79c2c3c4-df8e-41cc-8eaf-7f6b70834902" (UID: "79c2c3c4-df8e-41cc-8eaf-7f6b70834902"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:07:42.347598 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.347566    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.347598 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.347594    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.347598 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.347604    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.347832 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.347614    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-482vq\" (UniqueName: \"kubernetes.io/projected/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kube-api-access-482vq\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.347832 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.347624    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2c3c4-df8e-41cc-8eaf-7f6b70834902-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:07:42.525435 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.525352    4913 generic.go:356] "Generic (PLEG): container finished" podID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerID="5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435" exitCode=0
Oct 31 08:07:42.525435 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.525408    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"
Oct 31 08:07:42.525435 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.525423    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" event={"ID":"79c2c3c4-df8e-41cc-8eaf-7f6b70834902","Type":"ContainerDied","Data":"5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435"}
Oct 31 08:07:42.525719 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.525490    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw" event={"ID":"79c2c3c4-df8e-41cc-8eaf-7f6b70834902","Type":"ContainerDied","Data":"802de0e00deb8e5c9529cae004ea05576a8ef6280f89c061f1addfa53e7397d8"}
Oct 31 08:07:42.525719 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.525526    4913 scope.go:122] "RemoveContainer" containerID="22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66"
Oct 31 08:07:42.534435 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.534413    4913 scope.go:122] "RemoveContainer" containerID="5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435"
Oct 31 08:07:42.541670 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.541649    4913 scope.go:122] "RemoveContainer" containerID="26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011"
Oct 31 08:07:42.546375 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.546355    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"]
Oct 31 08:07:42.550037 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.549904    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-create-v2-predictor-694d996ccf-f4ncw"]
Oct 31 08:07:42.550037 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.549974    4913 scope.go:122] "RemoveContainer" containerID="22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66"
Oct 31 08:07:42.550286 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:07:42.550262    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66\": container with ID starting with 22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66 not found: ID does not exist" containerID="22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66"
Oct 31 08:07:42.550377 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.550291    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66"} err="failed to get container status \"22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66\": rpc error: code = NotFound desc = could not find container \"22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66\": container with ID starting with 22d05d159169274c1048bb45eec7519376f7f98397db2e54492feb4155a43b66 not found: ID does not exist"
Oct 31 08:07:42.550377 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.550310    4913 scope.go:122] "RemoveContainer" containerID="5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435"
Oct 31 08:07:42.550529 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:07:42.550514    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435\": container with ID starting with 5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435 not found: ID does not exist" containerID="5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435"
Oct 31 08:07:42.550569 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.550532    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435"} err="failed to get container status \"5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435\": rpc error: code = NotFound desc = could not find container \"5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435\": container with ID starting with 5319bcac2438ebd25bc0af9231ae7a5c04cbb84f583848bb9c2b08b6e311b435 not found: ID does not exist"
Oct 31 08:07:42.550569 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.550546    4913 scope.go:122] "RemoveContainer" containerID="26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011"
Oct 31 08:07:42.550806 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:07:42.550789    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011\": container with ID starting with 26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011 not found: ID does not exist" containerID="26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011"
Oct 31 08:07:42.550845 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:42.550812    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011"} err="failed to get container status \"26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011\": rpc error: code = NotFound desc = could not find container \"26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011\": container with ID starting with 26a61203e483ecff1654502f3a0a151c5f21ead06f0cb36822831ac35e7e0011 not found: ID does not exist"
Oct 31 08:07:43.418614 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.418584    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" path="/var/lib/kubelet/pods/79c2c3c4-df8e-41cc-8eaf-7f6b70834902/volumes"
Oct 31 08:07:43.649368 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649333    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"]
Oct 31 08:07:43.649678 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649664    4913 cpu_manager.go:414] "containerMap: removing container" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="storage-initializer"
Oct 31 08:07:43.649730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649679    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="storage-initializer"
Oct 31 08:07:43.649730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649690    4913 cpu_manager.go:414] "containerMap: removing container" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kube-rbac-proxy"
Oct 31 08:07:43.649730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649696    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kube-rbac-proxy"
Oct 31 08:07:43.649730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649707    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container"
Oct 31 08:07:43.649730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649712    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container"
Oct 31 08:07:43.649730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649725    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kube-rbac-proxy"
Oct 31 08:07:43.649730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649730    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kube-rbac-proxy"
Oct 31 08:07:43.649955 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649740    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="storage-initializer"
Oct 31 08:07:43.649955 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649745    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="storage-initializer"
Oct 31 08:07:43.649955 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649752    4913 cpu_manager.go:414] "containerMap: removing container" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container"
Oct 31 08:07:43.649955 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649757    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container"
Oct 31 08:07:43.649955 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649802    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kserve-container"
Oct 31 08:07:43.649955 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649811    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0799a12-056d-4863-8545-cf7f73513d3d" containerName="kube-rbac-proxy"
Oct 31 08:07:43.649955 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649818    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kserve-container"
Oct 31 08:07:43.649955 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.649826    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c2c3c4-df8e-41cc-8eaf-7f6b70834902" containerName="kube-rbac-proxy"
Oct 31 08:07:43.654151 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.654134    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.662157 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.662131    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:07:43.662157 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.662132    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 31 08:07:43.662157 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.662131    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:07:43.662553 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.662529    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:07:43.662709 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.662692    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 31 08:07:43.662795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.662780    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:07:43.662865 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.662845    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:07:43.663908 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.663879    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"]
Oct 31 08:07:43.696542 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.696514    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"]
Oct 31 08:07:43.700140 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.700120    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.703533 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.703515    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 31 08:07:43.711217 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.709781    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"]
Oct 31 08:07:43.759596 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759563    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec660f84-df03-4c72-b674-8b1bbd1bf55a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.759596 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759599    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.759779 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759622    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.759779 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759691    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.759779 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759725    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.759877 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759806    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qpd\" (UniqueName: \"kubernetes.io/projected/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-api-access-l9qpd\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.759877 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759829    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-proxy-tls\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.759942 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759890    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-cabundle-cert\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.759942 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759911    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.760004 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.759967    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnkv\" (UniqueName: \"kubernetes.io/projected/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-api-access-ncnkv\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.861288 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861249    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.861529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861299    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qpd\" (UniqueName: \"kubernetes.io/projected/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-api-access-l9qpd\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.861529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861320    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-proxy-tls\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.861529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861366    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-cabundle-cert\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.861529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861386    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.861529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861405    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnkv\" (UniqueName: \"kubernetes.io/projected/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-api-access-ncnkv\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.861816 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861631    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec660f84-df03-4c72-b674-8b1bbd1bf55a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.861816 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861678    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.861816 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861721    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.861816 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.861749    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.862160 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.862122    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.862160 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.862142    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.862378 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.862140    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.862378 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.862222    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.862469 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.862386    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.862469 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.862452    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-cabundle-cert\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.864197 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.864155    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec660f84-df03-4c72-b674-8b1bbd1bf55a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.864197 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.864155    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-proxy-tls\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.869383 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.869359    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnkv\" (UniqueName: \"kubernetes.io/projected/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-api-access-ncnkv\") pod \"isvc-canary-promote-predictor-6bbfb747bd-ss7jh\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:43.869956 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.869936    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qpd\" (UniqueName: \"kubernetes.io/projected/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-api-access-l9qpd\") pod \"isvc-canary-promote-v2-predictor-674dc9b765-n9qwr\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") " pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:43.977255 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:43.977145    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:44.018562 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:44.018522    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:44.251747 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:44.251720    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"]
Oct 31 08:07:44.253089 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:07:44.253064    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec660f84_df03_4c72_b674_8b1bbd1bf55a.slice/crio-1cf259e5d02a878eef5e1a1899a1f0a524019df441ce0e802ff637b5d75274a8 WatchSource:0}: Error finding container 1cf259e5d02a878eef5e1a1899a1f0a524019df441ce0e802ff637b5d75274a8: Status 404 returned error can't find the container with id 1cf259e5d02a878eef5e1a1899a1f0a524019df441ce0e802ff637b5d75274a8
Oct 31 08:07:44.307649 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:44.307621    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"]
Oct 31 08:07:44.310105 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:07:44.310071    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4855ce_e75a_4d6b_86b3_b6d00ada988f.slice/crio-a171a4415e40b8857dad0ffa359327611021b99c19aec59d444b0a5676331459 WatchSource:0}: Error finding container a171a4415e40b8857dad0ffa359327611021b99c19aec59d444b0a5676331459: Status 404 returned error can't find the container with id a171a4415e40b8857dad0ffa359327611021b99c19aec59d444b0a5676331459
Oct 31 08:07:44.537055 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:44.536953    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" event={"ID":"6f4855ce-e75a-4d6b-86b3-b6d00ada988f","Type":"ContainerStarted","Data":"a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27"}
Oct 31 08:07:44.537055 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:44.537029    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" event={"ID":"6f4855ce-e75a-4d6b-86b3-b6d00ada988f","Type":"ContainerStarted","Data":"a171a4415e40b8857dad0ffa359327611021b99c19aec59d444b0a5676331459"}
Oct 31 08:07:44.538673 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:44.538644    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" event={"ID":"ec660f84-df03-4c72-b674-8b1bbd1bf55a","Type":"ContainerStarted","Data":"d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7"}
Oct 31 08:07:44.538821 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:44.538697    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" event={"ID":"ec660f84-df03-4c72-b674-8b1bbd1bf55a","Type":"ContainerStarted","Data":"1cf259e5d02a878eef5e1a1899a1f0a524019df441ce0e802ff637b5d75274a8"}
Oct 31 08:07:48.557471 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:48.557435    4913 generic.go:356] "Generic (PLEG): container finished" podID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerID="d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7" exitCode=0
Oct 31 08:07:48.557918 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:48.557530    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" event={"ID":"ec660f84-df03-4c72-b674-8b1bbd1bf55a","Type":"ContainerDied","Data":"d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7"}
Oct 31 08:07:48.559287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:48.559264    4913 generic.go:356] "Generic (PLEG): container finished" podID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerID="a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27" exitCode=0
Oct 31 08:07:48.559382 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:48.559340    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" event={"ID":"6f4855ce-e75a-4d6b-86b3-b6d00ada988f","Type":"ContainerDied","Data":"a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27"}
Oct 31 08:07:49.398997 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:07:49.398959    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:07:49.566371 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:49.566334    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" event={"ID":"6f4855ce-e75a-4d6b-86b3-b6d00ada988f","Type":"ContainerStarted","Data":"2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0"}
Oct 31 08:07:49.566843 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:49.566410    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" event={"ID":"6f4855ce-e75a-4d6b-86b3-b6d00ada988f","Type":"ContainerStarted","Data":"683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6"}
Oct 31 08:07:49.566843 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:49.566624    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:49.568507 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:49.568481    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" event={"ID":"ec660f84-df03-4c72-b674-8b1bbd1bf55a","Type":"ContainerStarted","Data":"ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740"}
Oct 31 08:07:49.568640 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:49.568538    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" event={"ID":"ec660f84-df03-4c72-b674-8b1bbd1bf55a","Type":"ContainerStarted","Data":"68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d"}
Oct 31 08:07:49.568755 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:49.568738    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:49.585790 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:49.585745    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podStartSLOduration=2.444535734 podStartE2EDuration="6.585733813s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.141198079s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:07:49.5846711 +0000 UTC m=+2318.274439408" watchObservedRunningTime="2026-10-31 08:07:49.585733813 +0000 UTC m=+2318.275502105"
Oct 31 08:07:49.603166 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:49.603116    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podStartSLOduration=2.768234365 podStartE2EDuration="6.603105205s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83487084s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:07:49.601825318 +0000 UTC m=+2318.291593626" watchObservedRunningTime="2026-10-31 08:07:49.603105205 +0000 UTC m=+2318.292873497"
Oct 31 08:07:50.572815 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:50.572781    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:50.573219 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:50.572851    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:50.574067 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:50.574037    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:07:50.574067 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:50.574055    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:07:51.577334 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:51.577298    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:07:51.577755 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:51.577307    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:07:56.582178 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:56.582139    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:07:56.582701 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:56.582622    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:07:56.582701 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:56.582665    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:07:56.583062 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:07:56.583042    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:08:06.582679 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:06.582638    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:08:06.583089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:06.583067    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:08:16.583052 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:16.583009    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:08:16.583556 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:16.583009    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:08:19.399731 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:08:19.399695    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:08:26.583106 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:26.583064    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:08:26.583619 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:26.583064    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:08:36.582855 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:36.582818    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:08:36.583349 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:36.582959    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:08:46.582693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:46.582650    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:08:46.583179 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:46.583066    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:08:49.399124 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:08:49.399088    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:08:56.583353 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:56.583314    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:08:56.585669 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:08:56.583319    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:09:06.582967 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:06.582926    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:09:06.583390 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:06.583055    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:09:11.633404 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:11.633376    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:09:11.637958 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:11.637936    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:09:16.583681 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:16.583649    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:09:16.584078 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:16.583736    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:09:19.400000 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:09:19.399960    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:09:23.962227 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:23.962170    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"]
Oct 31 08:09:23.962685 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:23.962490    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" containerID="cri-o://683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6" gracePeriod=30
Oct 31 08:09:23.962871 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:23.962839    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kube-rbac-proxy" containerID="cri-o://2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0" gracePeriod=30
Oct 31 08:09:24.977085 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:24.977051    4913 generic.go:356] "Generic (PLEG): container finished" podID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerID="2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0" exitCode=0
Oct 31 08:09:24.977522 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:24.977110    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" event={"ID":"6f4855ce-e75a-4d6b-86b3-b6d00ada988f","Type":"ContainerDied","Data":"2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0"}
Oct 31 08:09:26.577700 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:26.577653    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.70:8643/healthz\": dial tcp 10.132.0.70:8643: connect: connection refused"
Oct 31 08:09:26.583319 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:26.583288    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.70:8080: connect: connection refused"
Oct 31 08:09:28.902351 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.902322    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:09:28.958152 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958121    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kserve-provision-location\") pod \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") "
Oct 31 08:09:28.958329 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958207    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-cabundle-cert\") pod \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") "
Oct 31 08:09:28.958329 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958257    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-api-access-ncnkv\" (UniqueName: \"kubernetes.io/projected/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-api-access-ncnkv\") pod \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") "
Oct 31 08:09:28.958329 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958322    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-proxy-tls\") pod \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") "
Oct 31 08:09:28.958449 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958375    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-rbac-proxy-sar-config\") pod \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\" (UID: \"6f4855ce-e75a-4d6b-86b3-b6d00ada988f\") "
Oct 31 08:09:28.958529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958494    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kserve-provision-location" pod "6f4855ce-e75a-4d6b-86b3-b6d00ada988f" (UID: "6f4855ce-e75a-4d6b-86b3-b6d00ada988f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:09:28.958633 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958606    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-cabundle-cert" pod "6f4855ce-e75a-4d6b-86b3-b6d00ada988f" (UID: "6f4855ce-e75a-4d6b-86b3-b6d00ada988f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:09:28.958725 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958645    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:28.958815 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.958794    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-rbac-proxy-sar-config" pod "6f4855ce-e75a-4d6b-86b3-b6d00ada988f" (UID: "6f4855ce-e75a-4d6b-86b3-b6d00ada988f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:09:28.960448 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.960421    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-proxy-tls" pod "6f4855ce-e75a-4d6b-86b3-b6d00ada988f" (UID: "6f4855ce-e75a-4d6b-86b3-b6d00ada988f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:09:28.960518 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.960466    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-api-access-ncnkv" pod "6f4855ce-e75a-4d6b-86b3-b6d00ada988f" (UID: "6f4855ce-e75a-4d6b-86b3-b6d00ada988f"). InnerVolumeSpecName "kube-api-access-ncnkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:09:28.994995 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.994920    4913 generic.go:356] "Generic (PLEG): container finished" podID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerID="683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6" exitCode=0
Oct 31 08:09:28.994995 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.994968    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" event={"ID":"6f4855ce-e75a-4d6b-86b3-b6d00ada988f","Type":"ContainerDied","Data":"683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6"}
Oct 31 08:09:28.994995 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.994976    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"
Oct 31 08:09:28.995224 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.995019    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh" event={"ID":"6f4855ce-e75a-4d6b-86b3-b6d00ada988f","Type":"ContainerDied","Data":"a171a4415e40b8857dad0ffa359327611021b99c19aec59d444b0a5676331459"}
Oct 31 08:09:28.995224 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:28.995048    4913 scope.go:122] "RemoveContainer" containerID="2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0"
Oct 31 08:09:29.004096 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.004074    4913 scope.go:122] "RemoveContainer" containerID="683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6"
Oct 31 08:09:29.011590 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.011570    4913 scope.go:122] "RemoveContainer" containerID="a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27"
Oct 31 08:09:29.016091 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.016070    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"]
Oct 31 08:09:29.019390 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.019371    4913 scope.go:122] "RemoveContainer" containerID="2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0"
Oct 31 08:09:29.019508 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.019493    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-predictor-6bbfb747bd-ss7jh"]
Oct 31 08:09:29.019693 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:09:29.019674    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0\": container with ID starting with 2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0 not found: ID does not exist" containerID="2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0"
Oct 31 08:09:29.019742 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.019701    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0"} err="failed to get container status \"2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0\": rpc error: code = NotFound desc = could not find container \"2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0\": container with ID starting with 2e128dc181a49a6b76f220bf55799fe8740df71172a4720eb665b16f7ea590b0 not found: ID does not exist"
Oct 31 08:09:29.019742 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.019719    4913 scope.go:122] "RemoveContainer" containerID="683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6"
Oct 31 08:09:29.019959 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:09:29.019941    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6\": container with ID starting with 683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6 not found: ID does not exist" containerID="683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6"
Oct 31 08:09:29.019998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.019968    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6"} err="failed to get container status \"683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6\": rpc error: code = NotFound desc = could not find container \"683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6\": container with ID starting with 683d14dc82ff1829d343eda520c1f487a259bcd2b456d1a3df182cebc09564c6 not found: ID does not exist"
Oct 31 08:09:29.019998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.019984    4913 scope.go:122] "RemoveContainer" containerID="a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27"
Oct 31 08:09:29.020259 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:09:29.020244    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27\": container with ID starting with a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27 not found: ID does not exist" containerID="a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27"
Oct 31 08:09:29.020322 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.020263    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27"} err="failed to get container status \"a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27\": rpc error: code = NotFound desc = could not find container \"a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27\": container with ID starting with a1050b10364061da09ea01247f7a9be78f2871572337e8ec9eef5a7d9133dc27 not found: ID does not exist"
Oct 31 08:09:29.060113 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.060084    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:29.060113 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.060113    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:29.060268 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.060124    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnkv\" (UniqueName: \"kubernetes.io/projected/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-kube-api-access-ncnkv\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:29.060268 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.060138    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4855ce-e75a-4d6b-86b3-b6d00ada988f-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:09:29.419669 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:09:29.419591    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" path="/var/lib/kubelet/pods/6f4855ce-e75a-4d6b-86b3-b6d00ada988f/volumes"
Oct 31 08:09:49.400399 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:09:49.400362    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:10:19.401020 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:10:19.400972    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:10:37.078437 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.078393    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"]
Oct 31 08:10:37.078941 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.078889    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kube-rbac-proxy"
Oct 31 08:10:37.078941 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.078910    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kube-rbac-proxy"
Oct 31 08:10:37.078941 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.078935    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="storage-initializer"
Oct 31 08:10:37.079107 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.078945    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="storage-initializer"
Oct 31 08:10:37.079107 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.078957    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container"
Oct 31 08:10:37.079107 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.078964    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container"
Oct 31 08:10:37.079107 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.079073    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kserve-container"
Oct 31 08:10:37.079107 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.079092    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4855ce-e75a-4d6b-86b3-b6d00ada988f" containerName="kube-rbac-proxy"
Oct 31 08:10:37.083979 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.083952    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.087960 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.087932    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Oct 31 08:10:37.087960 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.087939    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:10:37.088141 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.087997    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Oct 31 08:10:37.088794 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.088768    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:10:37.088936 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.088819    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:10:37.088936 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.088851    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:10:37.088936 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.088871    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:10:37.092023 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.092005    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"]
Oct 31 08:10:37.113558 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.113529    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.113705 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.113574    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.113705 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.113666    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.113705 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.113702    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcdbe17-737c-4cbe-ae0d-c769ff436722-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.113804 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.113722    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hw7h\" (UniqueName: \"kubernetes.io/projected/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-api-access-4hw7h\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.214892 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.214838    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.215080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.214908    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcdbe17-737c-4cbe-ae0d-c769ff436722-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.215080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.214935    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hw7h\" (UniqueName: \"kubernetes.io/projected/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-api-access-4hw7h\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.215080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.214984    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.215080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.215031    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.215414 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.215392    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.215730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.215708    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.215809 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.215748    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.217415 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.217397    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcdbe17-737c-4cbe-ae0d-c769ff436722-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.222454 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.222433    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hw7h\" (UniqueName: \"kubernetes.io/projected/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-api-access-4hw7h\") pod \"isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.404023 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.403931    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:10:37.681293 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:37.681268    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"]
Oct 31 08:10:38.274353 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:38.274318    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" event={"ID":"dfcdbe17-737c-4cbe-ae0d-c769ff436722","Type":"ContainerStarted","Data":"966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120"}
Oct 31 08:10:38.274742 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:38.274410    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" event={"ID":"dfcdbe17-737c-4cbe-ae0d-c769ff436722","Type":"ContainerStarted","Data":"26d8f996e83cdb26fc54c8ab318c9952818af35b4f73f440af9afe56b20d329d"}
Oct 31 08:10:42.292342 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:42.292309    4913 generic.go:356] "Generic (PLEG): container finished" podID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerID="966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120" exitCode=0
Oct 31 08:10:42.292747 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:42.292359    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" event={"ID":"dfcdbe17-737c-4cbe-ae0d-c769ff436722","Type":"ContainerDied","Data":"966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120"}
Oct 31 08:10:42.293691 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:10:42.293672    4913 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 08:10:49.405376 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:10:49.405324    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:11:00.379590 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:00.379556    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" event={"ID":"dfcdbe17-737c-4cbe-ae0d-c769ff436722","Type":"ContainerStarted","Data":"c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae"}
Oct 31 08:11:00.379988 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:00.379618    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" event={"ID":"dfcdbe17-737c-4cbe-ae0d-c769ff436722","Type":"ContainerStarted","Data":"60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026"}
Oct 31 08:11:00.379988 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:00.379803    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:11:00.398491 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:00.398442    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podStartSLOduration=1.974293919 podStartE2EDuration="23.398428931s" totalImagesPullingTime="17.597225363s" totalInitContainerRuntime="3.826909649s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:11:00.397180184 +0000 UTC m=+2509.086948494" watchObservedRunningTime="2026-10-31 08:11:00.398428931 +0000 UTC m=+2509.088197223"
Oct 31 08:11:01.383591 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:01.383554    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:11:01.384865 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:01.384837    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:11:02.387121 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:02.387074    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:11:04.450040 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:04.450005    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"]
Oct 31 08:11:04.450489 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:04.450302    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" containerID="cri-o://68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d" gracePeriod=30
Oct 31 08:11:04.450489 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:04.450365    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kube-rbac-proxy" containerID="cri-o://ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740" gracePeriod=30
Oct 31 08:11:05.401227 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:05.401172    4913 generic.go:356] "Generic (PLEG): container finished" podID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerID="ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740" exitCode=0
Oct 31 08:11:05.401227 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:05.401216    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" event={"ID":"ec660f84-df03-4c72-b674-8b1bbd1bf55a","Type":"ContainerDied","Data":"ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740"}
Oct 31 08:11:06.577743 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:06.577704    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": dial tcp 10.132.0.69:8643: connect: connection refused"
Oct 31 08:11:06.583295 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:06.583268    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Oct 31 08:11:07.391637 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:07.391610    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:11:07.392139 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:07.392114    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:11:09.390364 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.390338    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:11:09.419693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.419663    4913 generic.go:356] "Generic (PLEG): container finished" podID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerID="68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d" exitCode=0
Oct 31 08:11:09.419862 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.419796    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"
Oct 31 08:11:09.421147 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.421120    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" event={"ID":"ec660f84-df03-4c72-b674-8b1bbd1bf55a","Type":"ContainerDied","Data":"68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d"}
Oct 31 08:11:09.421283 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.421203    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr" event={"ID":"ec660f84-df03-4c72-b674-8b1bbd1bf55a","Type":"ContainerDied","Data":"1cf259e5d02a878eef5e1a1899a1f0a524019df441ce0e802ff637b5d75274a8"}
Oct 31 08:11:09.421283 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.421245    4913 scope.go:122] "RemoveContainer" containerID="ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740"
Oct 31 08:11:09.430077 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.430056    4913 scope.go:122] "RemoveContainer" containerID="68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d"
Oct 31 08:11:09.438169 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.438139    4913 scope.go:122] "RemoveContainer" containerID="d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7"
Oct 31 08:11:09.445543 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.445525    4913 scope.go:122] "RemoveContainer" containerID="ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740"
Oct 31 08:11:09.445808 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:11:09.445793    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740\": container with ID starting with ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740 not found: ID does not exist" containerID="ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740"
Oct 31 08:11:09.445853 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.445820    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740"} err="failed to get container status \"ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740\": rpc error: code = NotFound desc = could not find container \"ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740\": container with ID starting with ffe64d4c8b12874af62b922cdbcb69ea319dbf9c0383df7bd9eb2431d2fbd740 not found: ID does not exist"
Oct 31 08:11:09.445853 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.445837    4913 scope.go:122] "RemoveContainer" containerID="68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d"
Oct 31 08:11:09.446077 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:11:09.446057    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d\": container with ID starting with 68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d not found: ID does not exist" containerID="68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d"
Oct 31 08:11:09.446118 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.446084    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d"} err="failed to get container status \"68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d\": rpc error: code = NotFound desc = could not find container \"68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d\": container with ID starting with 68c4e559916ca616b47fe82ab5709bf7117ed71b584a3c947332e59c524ff00d not found: ID does not exist"
Oct 31 08:11:09.446118 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.446100    4913 scope.go:122] "RemoveContainer" containerID="d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7"
Oct 31 08:11:09.446368 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:11:09.446346    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7\": container with ID starting with d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7 not found: ID does not exist" containerID="d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7"
Oct 31 08:11:09.446421 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.446378    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7"} err="failed to get container status \"d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7\": rpc error: code = NotFound desc = could not find container \"d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7\": container with ID starting with d051c1a125bf77359a47916e5d035ed9aa7176436982016e5b8f8f2c59f04bb7 not found: ID does not exist"
Oct 31 08:11:09.510515 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.510432    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec660f84-df03-4c72-b674-8b1bbd1bf55a-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec660f84-df03-4c72-b674-8b1bbd1bf55a-proxy-tls\") pod \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") "
Oct 31 08:11:09.510515 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.510476    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-api-access-l9qpd\" (UniqueName: \"kubernetes.io/projected/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-api-access-l9qpd\") pod \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") "
Oct 31 08:11:09.510515 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.510510    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-cabundle-cert\") pod \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") "
Oct 31 08:11:09.510723 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.510594    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kserve-provision-location\") pod \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") "
Oct 31 08:11:09.510723 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.510611    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-rbac-proxy-sar-config\") pod \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\" (UID: \"ec660f84-df03-4c72-b674-8b1bbd1bf55a\") "
Oct 31 08:11:09.510957 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.510928    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-cabundle-cert" pod "ec660f84-df03-4c72-b674-8b1bbd1bf55a" (UID: "ec660f84-df03-4c72-b674-8b1bbd1bf55a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:11:09.511069 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.510975    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kserve-provision-location" pod "ec660f84-df03-4c72-b674-8b1bbd1bf55a" (UID: "ec660f84-df03-4c72-b674-8b1bbd1bf55a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:09.511069 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.511018    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-rbac-proxy-sar-config" pod "ec660f84-df03-4c72-b674-8b1bbd1bf55a" (UID: "ec660f84-df03-4c72-b674-8b1bbd1bf55a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:11:09.512665 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.512644    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec660f84-df03-4c72-b674-8b1bbd1bf55a-proxy-tls" pod "ec660f84-df03-4c72-b674-8b1bbd1bf55a" (UID: "ec660f84-df03-4c72-b674-8b1bbd1bf55a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:11:09.512734 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.512719    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-api-access-l9qpd" pod "ec660f84-df03-4c72-b674-8b1bbd1bf55a" (UID: "ec660f84-df03-4c72-b674-8b1bbd1bf55a"). InnerVolumeSpecName "kube-api-access-l9qpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:11:09.611390 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.611350    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9qpd\" (UniqueName: \"kubernetes.io/projected/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-api-access-l9qpd\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.611390 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.611383    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.611390 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.611398    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.611631 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.611410    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec660f84-df03-4c72-b674-8b1bbd1bf55a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.611631 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.611426    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec660f84-df03-4c72-b674-8b1bbd1bf55a-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:11:09.742480 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.742441    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"]
Oct 31 08:11:09.746402 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:09.746366    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-promote-v2-predictor-674dc9b765-n9qwr"]
Oct 31 08:11:11.421318 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:11.421282    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" path="/var/lib/kubelet/pods/ec660f84-df03-4c72-b674-8b1bbd1bf55a/volumes"
Oct 31 08:11:12.028974 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.028932    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"]
Oct 31 08:11:12.029457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.029437    4913 cpu_manager.go:414] "containerMap: removing container" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kube-rbac-proxy"
Oct 31 08:11:12.029548 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.029460    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kube-rbac-proxy"
Oct 31 08:11:12.029548 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.029506    4913 cpu_manager.go:414] "containerMap: removing container" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="storage-initializer"
Oct 31 08:11:12.029548 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.029515    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="storage-initializer"
Oct 31 08:11:12.029670 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.029556    4913 cpu_manager.go:414] "containerMap: removing container" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container"
Oct 31 08:11:12.029670 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.029565    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container"
Oct 31 08:11:12.029670 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.029668    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kserve-container"
Oct 31 08:11:12.029769 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.029685    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec660f84-df03-4c72-b674-8b1bbd1bf55a" containerName="kube-rbac-proxy"
Oct 31 08:11:12.034913 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.034890    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.041344 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.041317    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"]
Oct 31 08:11:12.067332 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.067301    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:11:12.067472 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.067402    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 31 08:11:12.067472 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.067419    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:11:12.085527 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.085499    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:11:12.085527 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.085519    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:11:12.085692 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.085602    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:11:12.085692 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.085605    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 31 08:11:12.126446 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.126357    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.126583 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.126450    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77373b75-3698-41e5-a19c-5fca2713f094-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.126583 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.126485    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.126686 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.126599    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77373b75-3698-41e5-a19c-5fca2713f094-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.126686 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.126648    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlcp4\" (UniqueName: \"kubernetes.io/projected/77373b75-3698-41e5-a19c-5fca2713f094-kube-api-access-jlcp4\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.227200 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.227146    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77373b75-3698-41e5-a19c-5fca2713f094-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.227365 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.227222    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlcp4\" (UniqueName: \"kubernetes.io/projected/77373b75-3698-41e5-a19c-5fca2713f094-kube-api-access-jlcp4\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.227365 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.227257    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.227365 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.227293    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77373b75-3698-41e5-a19c-5fca2713f094-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.227365 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.227322    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.227795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.227772    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77373b75-3698-41e5-a19c-5fca2713f094-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.228068 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.228049    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.228227 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.228204    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.229658 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.229642    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77373b75-3698-41e5-a19c-5fca2713f094-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.234780 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.234756    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlcp4\" (UniqueName: \"kubernetes.io/projected/77373b75-3698-41e5-a19c-5fca2713f094-kube-api-access-jlcp4\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-tlqd7\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.354728 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.354690    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:12.628151 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:12.628116    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"]
Oct 31 08:11:12.631235 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:11:12.631203    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77373b75_3698_41e5_a19c_5fca2713f094.slice/crio-d88ebc8e0e3dad03615f49eacfad7d9811e63170ec3ffa35a4223e17a870e776 WatchSource:0}: Error finding container d88ebc8e0e3dad03615f49eacfad7d9811e63170ec3ffa35a4223e17a870e776: Status 404 returned error can't find the container with id d88ebc8e0e3dad03615f49eacfad7d9811e63170ec3ffa35a4223e17a870e776
Oct 31 08:11:13.440352 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:13.440314    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" event={"ID":"77373b75-3698-41e5-a19c-5fca2713f094","Type":"ContainerStarted","Data":"fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a"}
Oct 31 08:11:13.440562 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:13.440385    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" event={"ID":"77373b75-3698-41e5-a19c-5fca2713f094","Type":"ContainerStarted","Data":"d88ebc8e0e3dad03615f49eacfad7d9811e63170ec3ffa35a4223e17a870e776"}
Oct 31 08:11:17.393000 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:17.392965    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:11:17.459042 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:17.459008    4913 generic.go:356] "Generic (PLEG): container finished" podID="77373b75-3698-41e5-a19c-5fca2713f094" containerID="fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a" exitCode=0
Oct 31 08:11:17.459246 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:17.459054    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" event={"ID":"77373b75-3698-41e5-a19c-5fca2713f094","Type":"ContainerDied","Data":"fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a"}
Oct 31 08:11:18.464823 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:18.464782    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" event={"ID":"77373b75-3698-41e5-a19c-5fca2713f094","Type":"ContainerStarted","Data":"087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e"}
Oct 31 08:11:18.465272 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:18.464851    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" event={"ID":"77373b75-3698-41e5-a19c-5fca2713f094","Type":"ContainerStarted","Data":"d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9"}
Oct 31 08:11:18.465272 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:18.465049    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:18.485160 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:18.485109    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podStartSLOduration=2.456574137 podStartE2EDuration="6.485096545s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.028522408s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:11:18.483532725 +0000 UTC m=+2527.173301025" watchObservedRunningTime="2026-10-31 08:11:18.485096545 +0000 UTC m=+2527.174864831"
Oct 31 08:11:19.399217 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:11:19.399161    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:11:19.468918 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:19.468887    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:19.470141 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:19.470112    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:11:20.473605 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:20.473565    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:11:25.478074 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:25.478027    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:11:25.478665 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:25.478645    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:11:27.392071 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:27.392034    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:11:35.478947 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:35.478910    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:11:37.392018 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:37.391979    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:11:45.478755 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:45.478710    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:11:47.392433 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:47.392393    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:11:49.400216 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:11:49.400165    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:11:55.479258 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:55.479142    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:11:57.392327 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:11:57.392289    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:12:05.478693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:05.478655    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:12:07.392246 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:07.392207    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.71:8080: connect: connection refused"
Oct 31 08:12:14.415128 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:14.415095    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:12:15.479176 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:15.479133    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:12:17.957538 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:17.957501    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"]
Oct 31 08:12:17.957913 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:17.957776    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container" containerID="cri-o://60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026" gracePeriod=30
Oct 31 08:12:17.958154 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:17.958125    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kube-rbac-proxy" containerID="cri-o://c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae" gracePeriod=30
Oct 31 08:12:18.714396 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:18.714361    4913 generic.go:356] "Generic (PLEG): container finished" podID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerID="c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae" exitCode=0
Oct 31 08:12:18.714547 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:18.714433    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" event={"ID":"dfcdbe17-737c-4cbe-ae0d-c769ff436722","Type":"ContainerDied","Data":"c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae"}
Oct 31 08:12:19.399831 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:12:19.399790    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:12:22.203243 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.203221    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:12:22.285275 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.285167    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-cabundle-cert\") pod \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") "
Oct 31 08:12:22.285275 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.285230    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-rbac-proxy-sar-config\") pod \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") "
Oct 31 08:12:22.285275 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.285263    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfcdbe17-737c-4cbe-ae0d-c769ff436722-proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcdbe17-737c-4cbe-ae0d-c769ff436722-proxy-tls\") pod \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") "
Oct 31 08:12:22.285584 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.285302    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kserve-provision-location\") pod \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") "
Oct 31 08:12:22.285584 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.285360    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-api-access-4hw7h\" (UniqueName: \"kubernetes.io/projected/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-api-access-4hw7h\") pod \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\" (UID: \"dfcdbe17-737c-4cbe-ae0d-c769ff436722\") "
Oct 31 08:12:22.285701 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.285676    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-cabundle-cert" pod "dfcdbe17-737c-4cbe-ae0d-c769ff436722" (UID: "dfcdbe17-737c-4cbe-ae0d-c769ff436722"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:12:22.285747 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.285687    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-rbac-proxy-sar-config" pod "dfcdbe17-737c-4cbe-ae0d-c769ff436722" (UID: "dfcdbe17-737c-4cbe-ae0d-c769ff436722"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:12:22.285747 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.285703    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kserve-provision-location" pod "dfcdbe17-737c-4cbe-ae0d-c769ff436722" (UID: "dfcdbe17-737c-4cbe-ae0d-c769ff436722"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:22.287546 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.287518    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfcdbe17-737c-4cbe-ae0d-c769ff436722-proxy-tls" pod "dfcdbe17-737c-4cbe-ae0d-c769ff436722" (UID: "dfcdbe17-737c-4cbe-ae0d-c769ff436722"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:12:22.287627 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.287523    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-api-access-4hw7h" pod "dfcdbe17-737c-4cbe-ae0d-c769ff436722" (UID: "dfcdbe17-737c-4cbe-ae0d-c769ff436722"). InnerVolumeSpecName "kube-api-access-4hw7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:12:22.386371 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.386336    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:22.386371 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.386365    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hw7h\" (UniqueName: \"kubernetes.io/projected/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-api-access-4hw7h\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:22.386371 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.386378    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:22.386607 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.386387    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfcdbe17-737c-4cbe-ae0d-c769ff436722-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:22.386607 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.386397    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfcdbe17-737c-4cbe-ae0d-c769ff436722-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:22.734947 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.734903    4913 generic.go:356] "Generic (PLEG): container finished" podID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerID="60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026" exitCode=0
Oct 31 08:12:22.735123 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.734973    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"
Oct 31 08:12:22.735123 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.734986    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" event={"ID":"dfcdbe17-737c-4cbe-ae0d-c769ff436722","Type":"ContainerDied","Data":"60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026"}
Oct 31 08:12:22.735123 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.735056    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs" event={"ID":"dfcdbe17-737c-4cbe-ae0d-c769ff436722","Type":"ContainerDied","Data":"26d8f996e83cdb26fc54c8ab318c9952818af35b4f73f440af9afe56b20d329d"}
Oct 31 08:12:22.735123 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.735082    4913 scope.go:122] "RemoveContainer" containerID="c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae"
Oct 31 08:12:22.743397 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.743377    4913 scope.go:122] "RemoveContainer" containerID="60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026"
Oct 31 08:12:22.750610 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.750593    4913 scope.go:122] "RemoveContainer" containerID="966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120"
Oct 31 08:12:22.756469 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.756441    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"]
Oct 31 08:12:22.758089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.758074    4913 scope.go:122] "RemoveContainer" containerID="c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae"
Oct 31 08:12:22.758433 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:12:22.758394    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae\": container with ID starting with c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae not found: ID does not exist" containerID="c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae"
Oct 31 08:12:22.758687 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.758501    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae"} err="failed to get container status \"c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae\": rpc error: code = NotFound desc = could not find container \"c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae\": container with ID starting with c3b93d32a424abb07d19c1aa60056b4ec896582e80f7800525cec41752cb59ae not found: ID does not exist"
Oct 31 08:12:22.758687 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.758546    4913 scope.go:122] "RemoveContainer" containerID="60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026"
Oct 31 08:12:22.759179 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:12:22.759057    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026\": container with ID starting with 60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026 not found: ID does not exist" containerID="60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026"
Oct 31 08:12:22.759179 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.759089    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026"} err="failed to get container status \"60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026\": rpc error: code = NotFound desc = could not find container \"60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026\": container with ID starting with 60c153a6f1b1189e8b70ece2bffbc571b62b5ded43fd80d49216f5536a1e8026 not found: ID does not exist"
Oct 31 08:12:22.759179 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.759109    4913 scope.go:122] "RemoveContainer" containerID="966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120"
Oct 31 08:12:22.759469 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:12:22.759449    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120\": container with ID starting with 966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120 not found: ID does not exist" containerID="966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120"
Oct 31 08:12:22.759558 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.759473    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120"} err="failed to get container status \"966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120\": rpc error: code = NotFound desc = could not find container \"966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120\": container with ID starting with 966b0cba149d3c60162e8a4455bafa4988b979a25cdd22f6f96faa8b39687120 not found: ID does not exist"
Oct 31 08:12:22.760218 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:22.760176    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-runtime-predictor-bf6dfb768-7c7rs"]
Oct 31 08:12:23.419819 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:23.419788    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" path="/var/lib/kubelet/pods/dfcdbe17-737c-4cbe-ae0d-c769ff436722/volumes"
Oct 31 08:12:25.479093 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:25.479058    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:12:28.414355 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:28.414308    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.72:8080: connect: connection refused"
Oct 31 08:12:37.836760 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:37.836722    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"]
Oct 31 08:12:37.837351 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:37.837046    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container" containerID="cri-o://d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9" gracePeriod=30
Oct 31 08:12:37.837351 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:37.837091    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kube-rbac-proxy" containerID="cri-o://087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e" gracePeriod=30
Oct 31 08:12:37.837974 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:37.837954    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:12:38.807759 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:38.807726    4913 generic.go:356] "Generic (PLEG): container finished" podID="77373b75-3698-41e5-a19c-5fca2713f094" containerID="087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e" exitCode=0
Oct 31 08:12:38.807933 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:38.807811    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" event={"ID":"77373b75-3698-41e5-a19c-5fca2713f094","Type":"ContainerDied","Data":"087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e"}
Oct 31 08:12:40.474021 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:40.473982    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.72:8643/healthz\": dial tcp 10.132.0.72:8643: connect: connection refused"
Oct 31 08:12:42.781145 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.781122    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:12:42.825608 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.825569    4913 generic.go:356] "Generic (PLEG): container finished" podID="77373b75-3698-41e5-a19c-5fca2713f094" containerID="d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9" exitCode=0
Oct 31 08:12:42.825757 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.825624    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"
Oct 31 08:12:42.825757 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.825627    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" event={"ID":"77373b75-3698-41e5-a19c-5fca2713f094","Type":"ContainerDied","Data":"d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9"}
Oct 31 08:12:42.825849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.825788    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7" event={"ID":"77373b75-3698-41e5-a19c-5fca2713f094","Type":"ContainerDied","Data":"d88ebc8e0e3dad03615f49eacfad7d9811e63170ec3ffa35a4223e17a870e776"}
Oct 31 08:12:42.825849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.825827    4913 scope.go:122] "RemoveContainer" containerID="087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e"
Oct 31 08:12:42.835805 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.835781    4913 scope.go:122] "RemoveContainer" containerID="d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9"
Oct 31 08:12:42.844107 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.844086    4913 scope.go:122] "RemoveContainer" containerID="fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a"
Oct 31 08:12:42.852601 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.852579    4913 scope.go:122] "RemoveContainer" containerID="087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e"
Oct 31 08:12:42.852923 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:12:42.852895    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e\": container with ID starting with 087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e not found: ID does not exist" containerID="087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e"
Oct 31 08:12:42.853017 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.852936    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e"} err="failed to get container status \"087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e\": rpc error: code = NotFound desc = could not find container \"087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e\": container with ID starting with 087d21a63146f30ded930588f326089d9210f40a0ebfece74e79a3afd28ce40e not found: ID does not exist"
Oct 31 08:12:42.853017 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.852957    4913 scope.go:122] "RemoveContainer" containerID="d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9"
Oct 31 08:12:42.853309 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:12:42.853287    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9\": container with ID starting with d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9 not found: ID does not exist" containerID="d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9"
Oct 31 08:12:42.853419 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.853315    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9"} err="failed to get container status \"d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9\": rpc error: code = NotFound desc = could not find container \"d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9\": container with ID starting with d7bd96bbf688864ecfb6bddc1e021128729f05641587d6040f9afb52ef76f2d9 not found: ID does not exist"
Oct 31 08:12:42.853419 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.853338    4913 scope.go:122] "RemoveContainer" containerID="fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a"
Oct 31 08:12:42.853603 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:12:42.853581    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a\": container with ID starting with fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a not found: ID does not exist" containerID="fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a"
Oct 31 08:12:42.853644 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.853611    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a"} err="failed to get container status \"fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a\": rpc error: code = NotFound desc = could not find container \"fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a\": container with ID starting with fb8d2c530ad7907765ef235838e754cfd4180d1f17f95276caf5da195e04cb8a not found: ID does not exist"
Oct 31 08:12:42.858872 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.858850    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77373b75-3698-41e5-a19c-5fca2713f094-proxy-tls\" (UniqueName: \"kubernetes.io/secret/77373b75-3698-41e5-a19c-5fca2713f094-proxy-tls\") pod \"77373b75-3698-41e5-a19c-5fca2713f094\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") "
Oct 31 08:12:42.858974 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.858891    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-kube-rbac-proxy-sar-config\") pod \"77373b75-3698-41e5-a19c-5fca2713f094\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") "
Oct 31 08:12:42.858974 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.858947    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77373b75-3698-41e5-a19c-5fca2713f094-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77373b75-3698-41e5-a19c-5fca2713f094-kserve-provision-location\") pod \"77373b75-3698-41e5-a19c-5fca2713f094\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") "
Oct 31 08:12:42.859099 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.858974    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-cabundle-cert\") pod \"77373b75-3698-41e5-a19c-5fca2713f094\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") "
Oct 31 08:12:42.859099 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.859025    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77373b75-3698-41e5-a19c-5fca2713f094-kube-api-access-jlcp4\" (UniqueName: \"kubernetes.io/projected/77373b75-3698-41e5-a19c-5fca2713f094-kube-api-access-jlcp4\") pod \"77373b75-3698-41e5-a19c-5fca2713f094\" (UID: \"77373b75-3698-41e5-a19c-5fca2713f094\") "
Oct 31 08:12:42.859449 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.859407    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77373b75-3698-41e5-a19c-5fca2713f094-kserve-provision-location" pod "77373b75-3698-41e5-a19c-5fca2713f094" (UID: "77373b75-3698-41e5-a19c-5fca2713f094"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:42.859449 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.859433    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-kube-rbac-proxy-sar-config" pod "77373b75-3698-41e5-a19c-5fca2713f094" (UID: "77373b75-3698-41e5-a19c-5fca2713f094"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:12:42.859572 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.859477    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-cabundle-cert" pod "77373b75-3698-41e5-a19c-5fca2713f094" (UID: "77373b75-3698-41e5-a19c-5fca2713f094"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:12:42.860993 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.860959    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77373b75-3698-41e5-a19c-5fca2713f094-proxy-tls" pod "77373b75-3698-41e5-a19c-5fca2713f094" (UID: "77373b75-3698-41e5-a19c-5fca2713f094"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:12:42.861328 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.861214    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77373b75-3698-41e5-a19c-5fca2713f094-kube-api-access-jlcp4" pod "77373b75-3698-41e5-a19c-5fca2713f094" (UID: "77373b75-3698-41e5-a19c-5fca2713f094"). InnerVolumeSpecName "kube-api-access-jlcp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:12:42.960495 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.960462    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77373b75-3698-41e5-a19c-5fca2713f094-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:42.960495 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.960490    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:42.960686 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.960503    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77373b75-3698-41e5-a19c-5fca2713f094-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:42.960686 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.960513    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77373b75-3698-41e5-a19c-5fca2713f094-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:42.960686 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:42.960522    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlcp4\" (UniqueName: \"kubernetes.io/projected/77373b75-3698-41e5-a19c-5fca2713f094-kube-api-access-jlcp4\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:12:43.163816 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:43.163785    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"]
Oct 31 08:12:43.172078 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:43.172041    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-tlqd7"]
Oct 31 08:12:43.419233 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:43.419159    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77373b75-3698-41e5-a19c-5fca2713f094" path="/var/lib/kubelet/pods/77373b75-3698-41e5-a19c-5fca2713f094/volumes"
Oct 31 08:12:49.399520 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:12:49.399480    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:12:52.232621 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232586    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"]
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232897    4913 cpu_manager.go:414] "containerMap: removing container" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232908    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232918    4913 cpu_manager.go:414] "containerMap: removing container" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="storage-initializer"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232923    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="storage-initializer"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232934    4913 cpu_manager.go:414] "containerMap: removing container" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232940    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232949    4913 cpu_manager.go:414] "containerMap: removing container" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="storage-initializer"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232954    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="storage-initializer"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232966    4913 cpu_manager.go:414] "containerMap: removing container" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kube-rbac-proxy"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232971    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kube-rbac-proxy"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232984    4913 cpu_manager.go:414] "containerMap: removing container" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kube-rbac-proxy"
Oct 31 08:12:52.233024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.232989    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kube-rbac-proxy"
Oct 31 08:12:52.233420 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.233035    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kube-rbac-proxy"
Oct 31 08:12:52.233420 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.233046    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcdbe17-737c-4cbe-ae0d-c769ff436722" containerName="kserve-container"
Oct 31 08:12:52.233420 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.233054    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kserve-container"
Oct 31 08:12:52.233420 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.233062    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="77373b75-3698-41e5-a19c-5fca2713f094" containerName="kube-rbac-proxy"
Oct 31 08:12:52.237857 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.237835    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.243711 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.243677    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 31 08:12:52.243833 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.243731    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:12:52.244758 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.244737    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:12:52.244860 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.244750    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:12:52.244860 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.244746    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:12:52.244860 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.244742    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:12:52.244860 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.244842    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 31 08:12:52.249507 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.249478    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"]
Oct 31 08:12:52.329620 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.329576    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c126e481-99e1-4555-b8ca-a15490f61f11-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.329620 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.329619    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.329846 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.329719    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.329846 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.329762    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c126e481-99e1-4555-b8ca-a15490f61f11-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.329846 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.329795    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6gsw\" (UniqueName: \"kubernetes.io/projected/c126e481-99e1-4555-b8ca-a15490f61f11-kube-api-access-z6gsw\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.430649 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.430605    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.430834 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.430659    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c126e481-99e1-4555-b8ca-a15490f61f11-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.430834 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.430687    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gsw\" (UniqueName: \"kubernetes.io/projected/c126e481-99e1-4555-b8ca-a15490f61f11-kube-api-access-z6gsw\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.430834 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.430720    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c126e481-99e1-4555-b8ca-a15490f61f11-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.430834 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.430737    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.431292 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.431239    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c126e481-99e1-4555-b8ca-a15490f61f11-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.431531 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.431508    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.431616 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.431526    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.433334 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.433311    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c126e481-99e1-4555-b8ca-a15490f61f11-proxy-tls\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.438475 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.438452    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gsw\" (UniqueName: \"kubernetes.io/projected/c126e481-99e1-4555-b8ca-a15490f61f11-kube-api-access-z6gsw\") pod \"isvc-canary-rollback-predictor-5f575bf5d4-lqxch\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") " pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.555826 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.555735    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:52.828482 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.828400    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"]
Oct 31 08:12:52.832333 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:12:52.832301    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc126e481_99e1_4555_b8ca_a15490f61f11.slice/crio-032719f683ba774a8c9b96928046cfa62568df64bb93d9ec56de0649e4fefc76 WatchSource:0}: Error finding container 032719f683ba774a8c9b96928046cfa62568df64bb93d9ec56de0649e4fefc76: Status 404 returned error can't find the container with id 032719f683ba774a8c9b96928046cfa62568df64bb93d9ec56de0649e4fefc76
Oct 31 08:12:52.871464 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:52.871397    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" event={"ID":"c126e481-99e1-4555-b8ca-a15490f61f11","Type":"ContainerStarted","Data":"032719f683ba774a8c9b96928046cfa62568df64bb93d9ec56de0649e4fefc76"}
Oct 31 08:12:53.877477 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:53.877442    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" event={"ID":"c126e481-99e1-4555-b8ca-a15490f61f11","Type":"ContainerStarted","Data":"79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33"}
Oct 31 08:12:57.895643 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:57.895609    4913 generic.go:356] "Generic (PLEG): container finished" podID="c126e481-99e1-4555-b8ca-a15490f61f11" containerID="79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33" exitCode=0
Oct 31 08:12:57.896062 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:57.895655    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" event={"ID":"c126e481-99e1-4555-b8ca-a15490f61f11","Type":"ContainerDied","Data":"79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33"}
Oct 31 08:12:58.901329 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:58.901294    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" event={"ID":"c126e481-99e1-4555-b8ca-a15490f61f11","Type":"ContainerStarted","Data":"9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece"}
Oct 31 08:12:58.901766 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:58.901359    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" event={"ID":"c126e481-99e1-4555-b8ca-a15490f61f11","Type":"ContainerStarted","Data":"2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2"}
Oct 31 08:12:58.901766 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:58.901551    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:58.920812 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:58.920761    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podStartSLOduration=2.750206251 podStartE2EDuration="6.920746027s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.170539776s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:12:58.919084805 +0000 UTC m=+2627.608853116" watchObservedRunningTime="2026-10-31 08:12:58.920746027 +0000 UTC m=+2627.610514319"
Oct 31 08:12:59.905744 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:59.905717    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:12:59.907134 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:12:59.907107    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:13:00.910325 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:00.910284    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:13:05.915239 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:05.915208    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:13:05.915771 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:05.915744    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:13:15.916625 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:15.916580    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:13:19.400036 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:13:19.399994    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:13:25.915942 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:25.915904    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:13:33.211661 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.211619    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"]
Oct 31 08:13:33.216691 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.216666    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.220791 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.220764    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:13:33.220791 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.220768    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:13:33.220983 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.220843    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:13:33.221044 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.221005    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Oct 31 08:13:33.221084 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.221045    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:13:33.221084 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.221073    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:13:33.221834 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.221816    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Oct 31 08:13:33.225362 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.225337    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"]
Oct 31 08:13:33.379279 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.379242    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.379452 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.379294    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.379452 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.379378    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.379452 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.379442    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.379557 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.379509    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkt5h\" (UniqueName: \"kubernetes.io/projected/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-api-access-zkt5h\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.480210 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.480117    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkt5h\" (UniqueName: \"kubernetes.io/projected/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-api-access-zkt5h\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.480210 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.480152    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.480210 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.480209    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.480505 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.480251    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.480505 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.480316    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.480851 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.480828    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.480940 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.480856    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.480997 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.480942    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.482606 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.482582    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-proxy-tls\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.487463 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.487444    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkt5h\" (UniqueName: \"kubernetes.io/projected/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-api-access-zkt5h\") pod \"isvc-xgboost-v2-predictor-7475869bcb-nhjft\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") " pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.536398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.536362    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:33.809749 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:33.809721    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"]
Oct 31 08:13:33.812324 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:13:33.812295    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a533dd_8bc6_4c6a_85d8_42b6625540e4.slice/crio-d5daecb2b932d9640cd8b003fcc8a57d7c2eb720c3d192667ddbcfa77bf0d006 WatchSource:0}: Error finding container d5daecb2b932d9640cd8b003fcc8a57d7c2eb720c3d192667ddbcfa77bf0d006: Status 404 returned error can't find the container with id d5daecb2b932d9640cd8b003fcc8a57d7c2eb720c3d192667ddbcfa77bf0d006
Oct 31 08:13:34.045420 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:34.045382    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" event={"ID":"c3a533dd-8bc6-4c6a-85d8-42b6625540e4","Type":"ContainerStarted","Data":"842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20"}
Oct 31 08:13:34.045634 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:34.045442    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" event={"ID":"c3a533dd-8bc6-4c6a-85d8-42b6625540e4","Type":"ContainerStarted","Data":"d5daecb2b932d9640cd8b003fcc8a57d7c2eb720c3d192667ddbcfa77bf0d006"}
Oct 31 08:13:35.915935 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:35.915896    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:13:38.062333 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:38.062298    4913 generic.go:356] "Generic (PLEG): container finished" podID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerID="842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20" exitCode=0
Oct 31 08:13:38.062688 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:38.062368    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" event={"ID":"c3a533dd-8bc6-4c6a-85d8-42b6625540e4","Type":"ContainerDied","Data":"842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20"}
Oct 31 08:13:39.069331 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:39.069294    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" event={"ID":"c3a533dd-8bc6-4c6a-85d8-42b6625540e4","Type":"ContainerStarted","Data":"fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15"}
Oct 31 08:13:39.069782 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:39.069354    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" event={"ID":"c3a533dd-8bc6-4c6a-85d8-42b6625540e4","Type":"ContainerStarted","Data":"1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb"}
Oct 31 08:13:39.069782 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:39.069593    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:39.089864 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:39.089814    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podStartSLOduration=1.940454206 podStartE2EDuration="6.089801634s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.149347428s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:13:39.087900843 +0000 UTC m=+2667.777669135" watchObservedRunningTime="2026-10-31 08:13:39.089801634 +0000 UTC m=+2667.779569925"
Oct 31 08:13:40.078120 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:40.078082    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:40.079294 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:40.079257    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:13:41.080835 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:41.080798    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:13:45.916102 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:45.916067    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:13:46.086048 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:46.086019    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:13:46.086556 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:46.086524    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:13:49.400507 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:13:49.400472    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:13:55.916435 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:55.916400    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:13:56.087128 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:13:56.087090    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:14:05.915814 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:05.915773    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:14:06.087430 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:06.087391    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:14:11.658367 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:11.658332    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:14:11.664754 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:11.664731    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:14:15.916229 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:15.916164    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.73:8080: connect: connection refused"
Oct 31 08:14:16.086482 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:16.086439    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:14:19.400035 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:14:19.399996    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:14:21.420902 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:21.420869    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:14:22.436738 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:22.436707    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"]
Oct 31 08:14:23.252656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:23.252611    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container" containerID="cri-o://2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2" gracePeriod=30
Oct 31 08:14:23.252656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:23.252630    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kube-rbac-proxy" containerID="cri-o://9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece" gracePeriod=30
Oct 31 08:14:24.258409 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:24.258369    4913 generic.go:356] "Generic (PLEG): container finished" podID="c126e481-99e1-4555-b8ca-a15490f61f11" containerID="9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece" exitCode=0
Oct 31 08:14:24.258799 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:24.258442    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" event={"ID":"c126e481-99e1-4555-b8ca-a15490f61f11","Type":"ContainerDied","Data":"9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece"}
Oct 31 08:14:25.910865 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:25.910822    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.73:8643/healthz\": dial tcp 10.132.0.73:8643: connect: connection refused"
Oct 31 08:14:26.087256 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:26.087219    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:14:28.194712 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.194686    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:14:28.276606 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.276521    4913 generic.go:356] "Generic (PLEG): container finished" podID="c126e481-99e1-4555-b8ca-a15490f61f11" containerID="2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2" exitCode=0
Oct 31 08:14:28.276606 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.276566    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" event={"ID":"c126e481-99e1-4555-b8ca-a15490f61f11","Type":"ContainerDied","Data":"2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2"}
Oct 31 08:14:28.276606 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.276584    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"
Oct 31 08:14:28.276824 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.276616    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch" event={"ID":"c126e481-99e1-4555-b8ca-a15490f61f11","Type":"ContainerDied","Data":"032719f683ba774a8c9b96928046cfa62568df64bb93d9ec56de0649e4fefc76"}
Oct 31 08:14:28.276824 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.276643    4913 scope.go:122] "RemoveContainer" containerID="9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece"
Oct 31 08:14:28.285157 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.285141    4913 scope.go:122] "RemoveContainer" containerID="2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2"
Oct 31 08:14:28.292710 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.292691    4913 scope.go:122] "RemoveContainer" containerID="79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33"
Oct 31 08:14:28.299698 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.299683    4913 scope.go:122] "RemoveContainer" containerID="9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece"
Oct 31 08:14:28.299958 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:14:28.299939    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece\": container with ID starting with 9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece not found: ID does not exist" containerID="9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece"
Oct 31 08:14:28.300010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.299973    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece"} err="failed to get container status \"9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece\": rpc error: code = NotFound desc = could not find container \"9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece\": container with ID starting with 9974e8638e11285f69da6405511e65aacb4c812d80e58ec2af769ec99c661ece not found: ID does not exist"
Oct 31 08:14:28.300010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.299991    4913 scope.go:122] "RemoveContainer" containerID="2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2"
Oct 31 08:14:28.300267 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:14:28.300244    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2\": container with ID starting with 2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2 not found: ID does not exist" containerID="2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2"
Oct 31 08:14:28.300386 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.300272    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2"} err="failed to get container status \"2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2\": rpc error: code = NotFound desc = could not find container \"2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2\": container with ID starting with 2eabe080ae295b516d1dead70302a00dcd6b3d8a8bd6f3286c38cdddf56650d2 not found: ID does not exist"
Oct 31 08:14:28.300386 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.300307    4913 scope.go:122] "RemoveContainer" containerID="79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33"
Oct 31 08:14:28.300549 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:14:28.300528    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33\": container with ID starting with 79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33 not found: ID does not exist" containerID="79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33"
Oct 31 08:14:28.300603 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.300556    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33"} err="failed to get container status \"79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33\": rpc error: code = NotFound desc = could not find container \"79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33\": container with ID starting with 79d204feb0d8281a015a64e86744542427b4b3e3185f12bd0f581dce00849e33 not found: ID does not exist"
Oct 31 08:14:28.341876 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.341849    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c126e481-99e1-4555-b8ca-a15490f61f11-kube-api-access-z6gsw\" (UniqueName: \"kubernetes.io/projected/c126e481-99e1-4555-b8ca-a15490f61f11-kube-api-access-z6gsw\") pod \"c126e481-99e1-4555-b8ca-a15490f61f11\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") "
Oct 31 08:14:28.342008 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.341892    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-kube-rbac-proxy-sar-config\") pod \"c126e481-99e1-4555-b8ca-a15490f61f11\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") "
Oct 31 08:14:28.342008 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.341933    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c126e481-99e1-4555-b8ca-a15490f61f11-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c126e481-99e1-4555-b8ca-a15490f61f11-kserve-provision-location\") pod \"c126e481-99e1-4555-b8ca-a15490f61f11\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") "
Oct 31 08:14:28.342008 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.341998    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c126e481-99e1-4555-b8ca-a15490f61f11-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c126e481-99e1-4555-b8ca-a15490f61f11-proxy-tls\") pod \"c126e481-99e1-4555-b8ca-a15490f61f11\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") "
Oct 31 08:14:28.342158 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.342043    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-cabundle-cert\") pod \"c126e481-99e1-4555-b8ca-a15490f61f11\" (UID: \"c126e481-99e1-4555-b8ca-a15490f61f11\") "
Oct 31 08:14:28.342331 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.342303    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c126e481-99e1-4555-b8ca-a15490f61f11-kserve-provision-location" pod "c126e481-99e1-4555-b8ca-a15490f61f11" (UID: "c126e481-99e1-4555-b8ca-a15490f61f11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:14:28.342434 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.342353    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-kube-rbac-proxy-sar-config" pod "c126e481-99e1-4555-b8ca-a15490f61f11" (UID: "c126e481-99e1-4555-b8ca-a15490f61f11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:14:28.342488 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.342427    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-cabundle-cert" pod "c126e481-99e1-4555-b8ca-a15490f61f11" (UID: "c126e481-99e1-4555-b8ca-a15490f61f11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:14:28.343980 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.343962    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c126e481-99e1-4555-b8ca-a15490f61f11-kube-api-access-z6gsw" pod "c126e481-99e1-4555-b8ca-a15490f61f11" (UID: "c126e481-99e1-4555-b8ca-a15490f61f11"). InnerVolumeSpecName "kube-api-access-z6gsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:14:28.344048 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.344009    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c126e481-99e1-4555-b8ca-a15490f61f11-proxy-tls" pod "c126e481-99e1-4555-b8ca-a15490f61f11" (UID: "c126e481-99e1-4555-b8ca-a15490f61f11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:14:28.443088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.443052    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c126e481-99e1-4555-b8ca-a15490f61f11-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:28.443088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.443082    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:28.443088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.443094    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6gsw\" (UniqueName: \"kubernetes.io/projected/c126e481-99e1-4555-b8ca-a15490f61f11-kube-api-access-z6gsw\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:28.443349 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.443106    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c126e481-99e1-4555-b8ca-a15490f61f11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:28.443349 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.443117    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c126e481-99e1-4555-b8ca-a15490f61f11-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:28.615963 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.615931    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"]
Oct 31 08:14:28.629674 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.629646    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-rollback-predictor-5f575bf5d4-lqxch"]
Oct 31 08:14:28.969683 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.969644    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"]
Oct 31 08:14:28.970056 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.970037    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="storage-initializer"
Oct 31 08:14:28.970137 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.970058    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="storage-initializer"
Oct 31 08:14:28.970137 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.970071    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container"
Oct 31 08:14:28.970137 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.970076    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container"
Oct 31 08:14:28.970137 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.970083    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kube-rbac-proxy"
Oct 31 08:14:28.970137 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.970088    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kube-rbac-proxy"
Oct 31 08:14:28.970340 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.970180    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kserve-container"
Oct 31 08:14:28.970340 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.970205    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" containerName="kube-rbac-proxy"
Oct 31 08:14:28.974750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.974722    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:28.978279 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.978258    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Oct 31 08:14:28.978364 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.978311    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:14:28.978364 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.978310    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:14:28.980843 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.980823    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 31 08:14:28.983724 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.983697    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:14:28.985766 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:28.985743    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"]
Oct 31 08:14:29.069768 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.069730    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"]
Oct 31 08:14:29.073496 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.073473    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.079020 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.079000    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 31 08:14:29.081035 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.081014    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:14:29.081035 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.081014    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:14:29.084138 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.084116    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"]
Oct 31 08:14:29.150827 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.150787    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/935bfe20-9a5f-4997-82dd-037a00b010fe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.150827 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.150826    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2pwg\" (UniqueName: \"kubernetes.io/projected/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-api-access-q2pwg\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.151075 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.150992    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.151075 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.151023    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.151162 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.151085    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935bfe20-9a5f-4997-82dd-037a00b010fe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.252202 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252094    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/935bfe20-9a5f-4997-82dd-037a00b010fe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.252202 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252136    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2pwg\" (UniqueName: \"kubernetes.io/projected/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-api-access-q2pwg\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.252724 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252223    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.252724 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252273    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.252724 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252345    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqhb\" (UniqueName: \"kubernetes.io/projected/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-api-access-dwqhb\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.252724 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252422    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.252724 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252443    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.252724 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252467    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-cabundle-cert\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.252724 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252489    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935bfe20-9a5f-4997-82dd-037a00b010fe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.253026 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252732    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57ce63c9-912b-463f-b90c-0af4e17fe9f1-proxy-tls\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.253026 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.252741    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/935bfe20-9a5f-4997-82dd-037a00b010fe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.253215 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.253171    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.253266 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.253177    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.254959 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.254935    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935bfe20-9a5f-4997-82dd-037a00b010fe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.263024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.263003    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2pwg\" (UniqueName: \"kubernetes.io/projected/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-api-access-q2pwg\") pod \"isvc-canary-stop-v2-predictor-79c977cff5-r45sp\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.292700 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.292673    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:29.357998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.356596    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.357998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.356676    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqhb\" (UniqueName: \"kubernetes.io/projected/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-api-access-dwqhb\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.357998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.356728    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-cabundle-cert\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.357998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.356762    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57ce63c9-912b-463f-b90c-0af4e17fe9f1-proxy-tls\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.357998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.356826    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.357998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.357108    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.357998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.357612    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.357998 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.357788    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-cabundle-cert\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.365656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.365614    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57ce63c9-912b-463f-b90c-0af4e17fe9f1-proxy-tls\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.366138 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.366120    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqhb\" (UniqueName: \"kubernetes.io/projected/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-api-access-dwqhb\") pod \"isvc-canary-stop-predictor-5485fd5dbd-gd8nj\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") " pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.391877 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.391849    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:29.420151 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.420114    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c126e481-99e1-4555-b8ca-a15490f61f11" path="/var/lib/kubelet/pods/c126e481-99e1-4555-b8ca-a15490f61f11/volumes"
Oct 31 08:14:29.574740 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.574658    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"]
Oct 31 08:14:29.577739 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:14:29.577708    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935bfe20_9a5f_4997_82dd_037a00b010fe.slice/crio-d8fdbdad919006df4ef059a00ded23bf69f58a7b7710686b2b26d3ee472d6492 WatchSource:0}: Error finding container d8fdbdad919006df4ef059a00ded23bf69f58a7b7710686b2b26d3ee472d6492: Status 404 returned error can't find the container with id d8fdbdad919006df4ef059a00ded23bf69f58a7b7710686b2b26d3ee472d6492
Oct 31 08:14:29.666380 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:29.666162    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"]
Oct 31 08:14:29.668889 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:14:29.668853    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ce63c9_912b_463f_b90c_0af4e17fe9f1.slice/crio-5a59bc03315fa1be8b5a64ddc8f3e2250f706ccebc3127a86b33475a22800168 WatchSource:0}: Error finding container 5a59bc03315fa1be8b5a64ddc8f3e2250f706ccebc3127a86b33475a22800168: Status 404 returned error can't find the container with id 5a59bc03315fa1be8b5a64ddc8f3e2250f706ccebc3127a86b33475a22800168
Oct 31 08:14:30.288263 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:30.288228    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" event={"ID":"57ce63c9-912b-463f-b90c-0af4e17fe9f1","Type":"ContainerStarted","Data":"3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662"}
Oct 31 08:14:30.288717 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:30.288287    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" event={"ID":"57ce63c9-912b-463f-b90c-0af4e17fe9f1","Type":"ContainerStarted","Data":"5a59bc03315fa1be8b5a64ddc8f3e2250f706ccebc3127a86b33475a22800168"}
Oct 31 08:14:30.290030 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:30.290002    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" event={"ID":"935bfe20-9a5f-4997-82dd-037a00b010fe","Type":"ContainerStarted","Data":"ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd"}
Oct 31 08:14:30.290147 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:30.290061    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" event={"ID":"935bfe20-9a5f-4997-82dd-037a00b010fe","Type":"ContainerStarted","Data":"d8fdbdad919006df4ef059a00ded23bf69f58a7b7710686b2b26d3ee472d6492"}
Oct 31 08:14:34.308581 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:34.308540    4913 generic.go:356] "Generic (PLEG): container finished" podID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerID="3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662" exitCode=0
Oct 31 08:14:34.309004 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:34.308612    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" event={"ID":"57ce63c9-912b-463f-b90c-0af4e17fe9f1","Type":"ContainerDied","Data":"3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662"}
Oct 31 08:14:34.310408 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:34.310343    4913 generic.go:356] "Generic (PLEG): container finished" podID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerID="ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd" exitCode=0
Oct 31 08:14:34.310408 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:34.310373    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" event={"ID":"935bfe20-9a5f-4997-82dd-037a00b010fe","Type":"ContainerDied","Data":"ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd"}
Oct 31 08:14:35.317126 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.317090    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" event={"ID":"57ce63c9-912b-463f-b90c-0af4e17fe9f1","Type":"ContainerStarted","Data":"7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b"}
Oct 31 08:14:35.317656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.317162    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" event={"ID":"57ce63c9-912b-463f-b90c-0af4e17fe9f1","Type":"ContainerStarted","Data":"40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e"}
Oct 31 08:14:35.317656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.317405    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:35.319167 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.319144    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" event={"ID":"935bfe20-9a5f-4997-82dd-037a00b010fe","Type":"ContainerStarted","Data":"dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553"}
Oct 31 08:14:35.319283 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.319209    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" event={"ID":"935bfe20-9a5f-4997-82dd-037a00b010fe","Type":"ContainerStarted","Data":"0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6"}
Oct 31 08:14:35.319548 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.319526    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:35.319585 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.319576    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:35.321112 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.321083    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:14:35.335968 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.335922    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podStartSLOduration=2.427816855 podStartE2EDuration="6.335909404s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.908092549s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:14:35.334421691 +0000 UTC m=+2724.024189984" watchObservedRunningTime="2026-10-31 08:14:35.335909404 +0000 UTC m=+2724.025677696"
Oct 31 08:14:35.352349 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:35.352300    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podStartSLOduration=3.249814574 podStartE2EDuration="7.352286322s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.102471748s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:14:35.350467542 +0000 UTC m=+2724.040235846" watchObservedRunningTime="2026-10-31 08:14:35.352286322 +0000 UTC m=+2724.042054613"
Oct 31 08:14:36.087345 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:36.087305    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:14:36.324594 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:36.324550    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:14:36.325099 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:36.324961    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:36.326216 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:36.326165    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:14:37.330148 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:37.330112    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:14:41.329523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:41.329477    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:14:41.330210 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:41.330155    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:14:42.334308 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:42.334276    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:14:42.334905 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:42.334881    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:14:46.087249 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:46.087210    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:14:47.419257 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:47.419230    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:14:49.400088 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:14:49.400046    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:14:51.330755 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:51.330720    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:14:52.335262 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:52.335223    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:14:54.246161 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:54.246065    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"]
Oct 31 08:14:54.246637 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:54.246380    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" containerID="cri-o://1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb" gracePeriod=30
Oct 31 08:14:54.246796 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:54.246767    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kube-rbac-proxy" containerID="cri-o://fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15" gracePeriod=30
Oct 31 08:14:54.401656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:54.401625    4913 generic.go:356] "Generic (PLEG): container finished" podID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerID="fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15" exitCode=0
Oct 31 08:14:54.401803 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:54.401679    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" event={"ID":"c3a533dd-8bc6-4c6a-85d8-42b6625540e4","Type":"ContainerDied","Data":"fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15"}
Oct 31 08:14:56.081465 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:56.081420    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.74:8643/healthz\": dial tcp 10.132.0.74:8643: connect: connection refused"
Oct 31 08:14:57.415114 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:57.415070    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.74:8080: connect: connection refused"
Oct 31 08:14:58.805120 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.805094    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:14:58.909741 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.909647    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-api-access-zkt5h\" (UniqueName: \"kubernetes.io/projected/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-api-access-zkt5h\") pod \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") "
Oct 31 08:14:58.909919 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.909752    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-proxy-tls\") pod \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") "
Oct 31 08:14:58.909919 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.909788    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-rbac-proxy-sar-config\") pod \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") "
Oct 31 08:14:58.909919 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.909823    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-cabundle-cert\") pod \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") "
Oct 31 08:14:58.909919 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.909871    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kserve-provision-location\") pod \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\" (UID: \"c3a533dd-8bc6-4c6a-85d8-42b6625540e4\") "
Oct 31 08:14:58.910292 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.910259    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-rbac-proxy-sar-config" pod "c3a533dd-8bc6-4c6a-85d8-42b6625540e4" (UID: "c3a533dd-8bc6-4c6a-85d8-42b6625540e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:14:58.910396 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.910285    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kserve-provision-location" pod "c3a533dd-8bc6-4c6a-85d8-42b6625540e4" (UID: "c3a533dd-8bc6-4c6a-85d8-42b6625540e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:14:58.910396 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.910328    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-cabundle-cert" pod "c3a533dd-8bc6-4c6a-85d8-42b6625540e4" (UID: "c3a533dd-8bc6-4c6a-85d8-42b6625540e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:14:58.911956 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.911931    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-proxy-tls" pod "c3a533dd-8bc6-4c6a-85d8-42b6625540e4" (UID: "c3a533dd-8bc6-4c6a-85d8-42b6625540e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:14:58.911956 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:58.911950    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-api-access-zkt5h" pod "c3a533dd-8bc6-4c6a-85d8-42b6625540e4" (UID: "c3a533dd-8bc6-4c6a-85d8-42b6625540e4"). InnerVolumeSpecName "kube-api-access-zkt5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:14:59.010840 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.010800    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:59.010840 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.010833    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:59.010840 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.010845    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:59.011091 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.010858    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:59.011091 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.010866    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkt5h\" (UniqueName: \"kubernetes.io/projected/c3a533dd-8bc6-4c6a-85d8-42b6625540e4-kube-api-access-zkt5h\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:14:59.426076 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.426041    4913 generic.go:356] "Generic (PLEG): container finished" podID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerID="1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb" exitCode=0
Oct 31 08:14:59.426286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.426121    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"
Oct 31 08:14:59.426286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.426137    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" event={"ID":"c3a533dd-8bc6-4c6a-85d8-42b6625540e4","Type":"ContainerDied","Data":"1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb"}
Oct 31 08:14:59.426286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.426237    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft" event={"ID":"c3a533dd-8bc6-4c6a-85d8-42b6625540e4","Type":"ContainerDied","Data":"d5daecb2b932d9640cd8b003fcc8a57d7c2eb720c3d192667ddbcfa77bf0d006"}
Oct 31 08:14:59.426286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.426281    4913 scope.go:122] "RemoveContainer" containerID="fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15"
Oct 31 08:14:59.437740 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.437683    4913 scope.go:122] "RemoveContainer" containerID="1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb"
Oct 31 08:14:59.441953 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.441930    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"]
Oct 31 08:14:59.445810 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.445787    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-xgboost-v2-predictor-7475869bcb-nhjft"]
Oct 31 08:14:59.447134 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.447120    4913 scope.go:122] "RemoveContainer" containerID="842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20"
Oct 31 08:14:59.454830 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.454806    4913 scope.go:122] "RemoveContainer" containerID="fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15"
Oct 31 08:14:59.455107 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:14:59.455088    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15\": container with ID starting with fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15 not found: ID does not exist" containerID="fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15"
Oct 31 08:14:59.455154 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.455119    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15"} err="failed to get container status \"fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15\": rpc error: code = NotFound desc = could not find container \"fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15\": container with ID starting with fa25aa5a0a7dba41d9b0834d420f6172aa43c7a5e7477a42db045ec6b222ad15 not found: ID does not exist"
Oct 31 08:14:59.455154 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.455139    4913 scope.go:122] "RemoveContainer" containerID="1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb"
Oct 31 08:14:59.455386 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:14:59.455367    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb\": container with ID starting with 1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb not found: ID does not exist" containerID="1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb"
Oct 31 08:14:59.455432 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.455394    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb"} err="failed to get container status \"1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb\": rpc error: code = NotFound desc = could not find container \"1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb\": container with ID starting with 1ecf76d0bbd855a199c2f6bf06638ea8fc9761554af7f286fcf888c57d9cdfdb not found: ID does not exist"
Oct 31 08:14:59.455432 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.455413    4913 scope.go:122] "RemoveContainer" containerID="842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20"
Oct 31 08:14:59.455667 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:14:59.455650    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20\": container with ID starting with 842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20 not found: ID does not exist" containerID="842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20"
Oct 31 08:14:59.455708 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:14:59.455674    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20"} err="failed to get container status \"842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20\": rpc error: code = NotFound desc = could not find container \"842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20\": container with ID starting with 842f6da969ca9d99b99df5f294e9ae788db7a16e0c57e24a1260b53cda77eb20 not found: ID does not exist"
Oct 31 08:15:01.330696 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:01.330653    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:15:01.418814 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:01.418773    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" path="/var/lib/kubelet/pods/c3a533dd-8bc6-4c6a-85d8-42b6625540e4/volumes"
Oct 31 08:15:02.335171 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:02.335130    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:15:11.330609 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:11.330570    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:15:12.334837 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:12.334793    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:15:19.400027 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:15:19.399988    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:15:21.330127 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:21.330088    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:15:22.335524 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:22.335487    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:15:31.330932 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:31.330894    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:15:32.335610 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:32.335572    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:15:41.330420 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:41.330378    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:15:42.334975 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:42.334937    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:15:49.400948 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:15:49.400908    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:15:51.330357 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:51.330313    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.75:8080: connect: connection refused"
Oct 31 08:15:51.415965 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:51.415927    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:15:58.415084 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:15:58.415032    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:16:01.418785 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:01.418758    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:16:09.154075 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.154043    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"]
Oct 31 08:16:09.154544 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.154292    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container" containerID="cri-o://0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6" gracePeriod=30
Oct 31 08:16:09.154544 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.154322    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kube-rbac-proxy" containerID="cri-o://dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553" gracePeriod=30
Oct 31 08:16:09.199530 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.199499    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"]
Oct 31 08:16:09.199755 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.199729    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" containerID="cri-o://40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e" gracePeriod=30
Oct 31 08:16:09.199885 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.199761    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kube-rbac-proxy" containerID="cri-o://7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b" gracePeriod=30
Oct 31 08:16:09.717705 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.717666    4913 generic.go:356] "Generic (PLEG): container finished" podID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerID="7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b" exitCode=0
Oct 31 08:16:09.717896 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.717739    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" event={"ID":"57ce63c9-912b-463f-b90c-0af4e17fe9f1","Type":"ContainerDied","Data":"7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b"}
Oct 31 08:16:09.719677 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.719657    4913 generic.go:356] "Generic (PLEG): container finished" podID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerID="dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553" exitCode=0
Oct 31 08:16:09.719785 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:09.719683    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" event={"ID":"935bfe20-9a5f-4997-82dd-037a00b010fe","Type":"ContainerDied","Data":"dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553"}
Oct 31 08:16:11.325591 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:11.325543    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.75:8643/healthz\": dial tcp 10.132.0.75:8643: connect: connection refused"
Oct 31 08:16:11.416356 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:11.416316    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.76:8080: connect: connection refused"
Oct 31 08:16:12.331407 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:12.331363    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.76:8643/healthz\": dial tcp 10.132.0.76:8643: connect: connection refused"
Oct 31 08:16:14.322529 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.322507    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:16:14.423010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.422982    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-cabundle-cert\") pod \"935bfe20-9a5f-4997-82dd-037a00b010fe\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") "
Oct 31 08:16:14.423143 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423053    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935bfe20-9a5f-4997-82dd-037a00b010fe-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/935bfe20-9a5f-4997-82dd-037a00b010fe-kserve-provision-location\") pod \"935bfe20-9a5f-4997-82dd-037a00b010fe\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") "
Oct 31 08:16:14.423143 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423096    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-api-access-q2pwg\" (UniqueName: \"kubernetes.io/projected/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-api-access-q2pwg\") pod \"935bfe20-9a5f-4997-82dd-037a00b010fe\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") "
Oct 31 08:16:14.423336 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423165    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-rbac-proxy-sar-config\") pod \"935bfe20-9a5f-4997-82dd-037a00b010fe\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") "
Oct 31 08:16:14.423336 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423209    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/935bfe20-9a5f-4997-82dd-037a00b010fe-proxy-tls\" (UniqueName: \"kubernetes.io/secret/935bfe20-9a5f-4997-82dd-037a00b010fe-proxy-tls\") pod \"935bfe20-9a5f-4997-82dd-037a00b010fe\" (UID: \"935bfe20-9a5f-4997-82dd-037a00b010fe\") "
Oct 31 08:16:14.423433 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423341    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935bfe20-9a5f-4997-82dd-037a00b010fe-kserve-provision-location" pod "935bfe20-9a5f-4997-82dd-037a00b010fe" (UID: "935bfe20-9a5f-4997-82dd-037a00b010fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:16:14.423433 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423367    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-cabundle-cert" pod "935bfe20-9a5f-4997-82dd-037a00b010fe" (UID: "935bfe20-9a5f-4997-82dd-037a00b010fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:16:14.423543 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423457    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.423543 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423477    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/935bfe20-9a5f-4997-82dd-037a00b010fe-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.423643 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.423608    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-rbac-proxy-sar-config" pod "935bfe20-9a5f-4997-82dd-037a00b010fe" (UID: "935bfe20-9a5f-4997-82dd-037a00b010fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:16:14.425284 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.425258    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935bfe20-9a5f-4997-82dd-037a00b010fe-proxy-tls" pod "935bfe20-9a5f-4997-82dd-037a00b010fe" (UID: "935bfe20-9a5f-4997-82dd-037a00b010fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:16:14.425390 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.425348    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-api-access-q2pwg" pod "935bfe20-9a5f-4997-82dd-037a00b010fe" (UID: "935bfe20-9a5f-4997-82dd-037a00b010fe"). InnerVolumeSpecName "kube-api-access-q2pwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:16:14.431893 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.431875    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:16:14.524206 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524148    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57ce63c9-912b-463f-b90c-0af4e17fe9f1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/57ce63c9-912b-463f-b90c-0af4e17fe9f1-proxy-tls\") pod \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") "
Oct 31 08:16:14.524398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524253    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-cabundle-cert\") pod \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") "
Oct 31 08:16:14.524398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524292    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-api-access-dwqhb\" (UniqueName: \"kubernetes.io/projected/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-api-access-dwqhb\") pod \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") "
Oct 31 08:16:14.524398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524333    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-rbac-proxy-sar-config\") pod \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") "
Oct 31 08:16:14.524398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524372    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kserve-provision-location\") pod \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\" (UID: \"57ce63c9-912b-463f-b90c-0af4e17fe9f1\") "
Oct 31 08:16:14.524630 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524580    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.524630 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524599    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935bfe20-9a5f-4997-82dd-037a00b010fe-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.524630 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524615    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2pwg\" (UniqueName: \"kubernetes.io/projected/935bfe20-9a5f-4997-82dd-037a00b010fe-kube-api-access-q2pwg\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.524742 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524700    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-cabundle-cert" pod "57ce63c9-912b-463f-b90c-0af4e17fe9f1" (UID: "57ce63c9-912b-463f-b90c-0af4e17fe9f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:16:14.524780 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524739    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-rbac-proxy-sar-config" pod "57ce63c9-912b-463f-b90c-0af4e17fe9f1" (UID: "57ce63c9-912b-463f-b90c-0af4e17fe9f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:16:14.524780 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.524755    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kserve-provision-location" pod "57ce63c9-912b-463f-b90c-0af4e17fe9f1" (UID: "57ce63c9-912b-463f-b90c-0af4e17fe9f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:16:14.526388 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.526365    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ce63c9-912b-463f-b90c-0af4e17fe9f1-proxy-tls" pod "57ce63c9-912b-463f-b90c-0af4e17fe9f1" (UID: "57ce63c9-912b-463f-b90c-0af4e17fe9f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:16:14.526478 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.526415    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-api-access-dwqhb" pod "57ce63c9-912b-463f-b90c-0af4e17fe9f1" (UID: "57ce63c9-912b-463f-b90c-0af4e17fe9f1"). InnerVolumeSpecName "kube-api-access-dwqhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:16:14.625676 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.625643    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.625676 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.625670    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwqhb\" (UniqueName: \"kubernetes.io/projected/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-api-access-dwqhb\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.625676 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.625681    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.625903 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.625690    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ce63c9-912b-463f-b90c-0af4e17fe9f1-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.625903 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.625699    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57ce63c9-912b-463f-b90c-0af4e17fe9f1-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:16:14.748762 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.748672    4913 generic.go:356] "Generic (PLEG): container finished" podID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerID="40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e" exitCode=0
Oct 31 08:16:14.748762 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.748729    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"
Oct 31 08:16:14.748947 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.748750    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" event={"ID":"57ce63c9-912b-463f-b90c-0af4e17fe9f1","Type":"ContainerDied","Data":"40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e"}
Oct 31 08:16:14.748947 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.748819    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj" event={"ID":"57ce63c9-912b-463f-b90c-0af4e17fe9f1","Type":"ContainerDied","Data":"5a59bc03315fa1be8b5a64ddc8f3e2250f706ccebc3127a86b33475a22800168"}
Oct 31 08:16:14.748947 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.748862    4913 scope.go:122] "RemoveContainer" containerID="7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b"
Oct 31 08:16:14.750742 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.750718    4913 generic.go:356] "Generic (PLEG): container finished" podID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerID="0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6" exitCode=0
Oct 31 08:16:14.750890 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.750775    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"
Oct 31 08:16:14.750890 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.750803    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" event={"ID":"935bfe20-9a5f-4997-82dd-037a00b010fe","Type":"ContainerDied","Data":"0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6"}
Oct 31 08:16:14.750890 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.750847    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp" event={"ID":"935bfe20-9a5f-4997-82dd-037a00b010fe","Type":"ContainerDied","Data":"d8fdbdad919006df4ef059a00ded23bf69f58a7b7710686b2b26d3ee472d6492"}
Oct 31 08:16:14.758358 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.757939    4913 scope.go:122] "RemoveContainer" containerID="40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e"
Oct 31 08:16:14.765926 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.765908    4913 scope.go:122] "RemoveContainer" containerID="3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662"
Oct 31 08:16:14.770440 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.770420    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"]
Oct 31 08:16:14.776901 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.776880    4913 scope.go:122] "RemoveContainer" containerID="7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b"
Oct 31 08:16:14.777357 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:16:14.777338    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b\": container with ID starting with 7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b not found: ID does not exist" containerID="7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b"
Oct 31 08:16:14.777422 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.777367    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b"} err="failed to get container status \"7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b\": rpc error: code = NotFound desc = could not find container \"7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b\": container with ID starting with 7aa91c17de84d7f837af9424a0b2bc27290316caa9c03b5214eb5c738366a99b not found: ID does not exist"
Oct 31 08:16:14.777422 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.777385    4913 scope.go:122] "RemoveContainer" containerID="40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e"
Oct 31 08:16:14.777667 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:16:14.777646    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e\": container with ID starting with 40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e not found: ID does not exist" containerID="40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e"
Oct 31 08:16:14.777713 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.777675    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e"} err="failed to get container status \"40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e\": rpc error: code = NotFound desc = could not find container \"40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e\": container with ID starting with 40f19c937a851776a4aec8cb81e1e7772b0a7422c032595079b91f22b808a15e not found: ID does not exist"
Oct 31 08:16:14.777713 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.777692    4913 scope.go:122] "RemoveContainer" containerID="3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662"
Oct 31 08:16:14.777942 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:16:14.777916    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662\": container with ID starting with 3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662 not found: ID does not exist" containerID="3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662"
Oct 31 08:16:14.778009 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.777951    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662"} err="failed to get container status \"3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662\": rpc error: code = NotFound desc = could not find container \"3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662\": container with ID starting with 3c323b75f8a02fe4a54bb28c3b537f0e6b016836312ab9789ab324405e39c662 not found: ID does not exist"
Oct 31 08:16:14.778009 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.777972    4913 scope.go:122] "RemoveContainer" containerID="dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553"
Oct 31 08:16:14.778009 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.777991    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-predictor-5485fd5dbd-gd8nj"]
Oct 31 08:16:14.785278 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.785258    4913 scope.go:122] "RemoveContainer" containerID="0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6"
Oct 31 08:16:14.788410 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.788390    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"]
Oct 31 08:16:14.792862 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.792840    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-canary-stop-v2-predictor-79c977cff5-r45sp"]
Oct 31 08:16:14.793063 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.793049    4913 scope.go:122] "RemoveContainer" containerID="ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd"
Oct 31 08:16:14.799979 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.799959    4913 scope.go:122] "RemoveContainer" containerID="dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553"
Oct 31 08:16:14.800274 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:16:14.800252    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553\": container with ID starting with dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553 not found: ID does not exist" containerID="dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553"
Oct 31 08:16:14.800333 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.800282    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553"} err="failed to get container status \"dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553\": rpc error: code = NotFound desc = could not find container \"dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553\": container with ID starting with dd2b7edd79cbaf29276b89d623c8bdaaee1325cf7b68f7d7ba88de9407b01553 not found: ID does not exist"
Oct 31 08:16:14.800333 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.800303    4913 scope.go:122] "RemoveContainer" containerID="0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6"
Oct 31 08:16:14.800576 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:16:14.800556    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6\": container with ID starting with 0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6 not found: ID does not exist" containerID="0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6"
Oct 31 08:16:14.800638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.800587    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6"} err="failed to get container status \"0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6\": rpc error: code = NotFound desc = could not find container \"0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6\": container with ID starting with 0de493b32d2bcc6e77ac5de3f5bee298cf37e5e80cff8fd46610f024103a11e6 not found: ID does not exist"
Oct 31 08:16:14.800638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.800611    4913 scope.go:122] "RemoveContainer" containerID="ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd"
Oct 31 08:16:14.800845 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:16:14.800828    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd\": container with ID starting with ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd not found: ID does not exist" containerID="ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd"
Oct 31 08:16:14.800892 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:14.800852    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd"} err="failed to get container status \"ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd\": rpc error: code = NotFound desc = could not find container \"ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd\": container with ID starting with ffa5fd06d5f9eea8475e9f096ae04eeeca9c6199e0ea9b4172d601a8614c4ccd not found: ID does not exist"
Oct 31 08:16:15.419115 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:15.419041    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" path="/var/lib/kubelet/pods/57ce63c9-912b-463f-b90c-0af4e17fe9f1/volumes"
Oct 31 08:16:15.419601 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:15.419584    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" path="/var/lib/kubelet/pods/935bfe20-9a5f-4997-82dd-037a00b010fe/volumes"
Oct 31 08:16:16.735004 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.734969    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"]
Oct 31 08:16:16.735495 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735479    4913 cpu_manager.go:414] "containerMap: removing container" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="storage-initializer"
Oct 31 08:16:16.735562 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735497    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="storage-initializer"
Oct 31 08:16:16.735562 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735524    4913 cpu_manager.go:414] "containerMap: removing container" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kube-rbac-proxy"
Oct 31 08:16:16.735562 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735533    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kube-rbac-proxy"
Oct 31 08:16:16.735562 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735554    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="storage-initializer"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735562    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="storage-initializer"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735580    4913 cpu_manager.go:414] "containerMap: removing container" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="storage-initializer"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735589    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="storage-initializer"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735601    4913 cpu_manager.go:414] "containerMap: removing container" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kube-rbac-proxy"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735609    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kube-rbac-proxy"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735621    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kube-rbac-proxy"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735628    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kube-rbac-proxy"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735640    4913 cpu_manager.go:414] "containerMap: removing container" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735648    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735662    4913 cpu_manager.go:414] "containerMap: removing container" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735669    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735680    4913 cpu_manager.go:414] "containerMap: removing container" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735687    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container"
Oct 31 08:16:16.735767 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735768    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kube-rbac-proxy"
Oct 31 08:16:16.736453 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735783    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kube-rbac-proxy"
Oct 31 08:16:16.736453 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735792    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kube-rbac-proxy"
Oct 31 08:16:16.736453 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735800    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="935bfe20-9a5f-4997-82dd-037a00b010fe" containerName="kserve-container"
Oct 31 08:16:16.736453 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735809    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a533dd-8bc6-4c6a-85d8-42b6625540e4" containerName="kserve-container"
Oct 31 08:16:16.736453 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.735819    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ce63c9-912b-463f-b90c-0af4e17fe9f1" containerName="kserve-container"
Oct 31 08:16:16.741048 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.741023    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.746663 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.746634    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:16:16.747388 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.747365    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-lightgbm-predictor-serving-cert\""
Oct 31 08:16:16.747656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.747523    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:16:16.747656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.747555    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:16:16.747656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.747575    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Oct 31 08:16:16.747656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.747619    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:16:16.747656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.747625    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:16:16.748771 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.748749    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"]
Oct 31 08:16:16.842344 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.842314    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e755559-c934-42e0-8def-b8ee970d10d3-proxy-tls\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.842344 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.842347    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.842567 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.842381    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5n5\" (UniqueName: \"kubernetes.io/projected/4e755559-c934-42e0-8def-b8ee970d10d3-kube-api-access-7f5n5\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.842567 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.842554    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e755559-c934-42e0-8def-b8ee970d10d3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.842642 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.842613    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-cabundle-cert\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.943595 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.943561    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5n5\" (UniqueName: \"kubernetes.io/projected/4e755559-c934-42e0-8def-b8ee970d10d3-kube-api-access-7f5n5\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.943775 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.943605    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e755559-c934-42e0-8def-b8ee970d10d3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.943775 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.943646    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-cabundle-cert\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.943775 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.943671    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e755559-c934-42e0-8def-b8ee970d10d3-proxy-tls\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.943775 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.943687    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.944025 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.944002    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e755559-c934-42e0-8def-b8ee970d10d3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.944308 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.944285    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-cabundle-cert\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.944432 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.944412    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.946210 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.946175    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e755559-c934-42e0-8def-b8ee970d10d3-proxy-tls\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:16.951509 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:16.951485    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5n5\" (UniqueName: \"kubernetes.io/projected/4e755559-c934-42e0-8def-b8ee970d10d3-kube-api-access-7f5n5\") pod \"isvc-lightgbm-predictor-5b4d64fbc6-hrvtb\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:17.063321 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:17.063230    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:17.336834 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:17.336802    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"]
Oct 31 08:16:17.339531 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:16:17.339498    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e755559_c934_42e0_8def_b8ee970d10d3.slice/crio-9d085dcd3deff04db97d892b9bf34add57d286d9243b8ead74026dcd4f6ef421 WatchSource:0}: Error finding container 9d085dcd3deff04db97d892b9bf34add57d286d9243b8ead74026dcd4f6ef421: Status 404 returned error can't find the container with id 9d085dcd3deff04db97d892b9bf34add57d286d9243b8ead74026dcd4f6ef421
Oct 31 08:16:17.767625 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:17.767587    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" event={"ID":"4e755559-c934-42e0-8def-b8ee970d10d3","Type":"ContainerStarted","Data":"71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d"}
Oct 31 08:16:17.768033 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:17.767649    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" event={"ID":"4e755559-c934-42e0-8def-b8ee970d10d3","Type":"ContainerStarted","Data":"9d085dcd3deff04db97d892b9bf34add57d286d9243b8ead74026dcd4f6ef421"}
Oct 31 08:16:19.399082 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:16:19.399043    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:16:21.787035 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:21.786952    4913 generic.go:356] "Generic (PLEG): container finished" podID="4e755559-c934-42e0-8def-b8ee970d10d3" containerID="71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d" exitCode=0
Oct 31 08:16:21.787035 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:21.787024    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" event={"ID":"4e755559-c934-42e0-8def-b8ee970d10d3","Type":"ContainerDied","Data":"71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d"}
Oct 31 08:16:21.788366 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:21.788349    4913 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 08:16:28.825251 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:28.825211    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" event={"ID":"4e755559-c934-42e0-8def-b8ee970d10d3","Type":"ContainerStarted","Data":"144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9"}
Oct 31 08:16:28.825799 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:28.825272    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" event={"ID":"4e755559-c934-42e0-8def-b8ee970d10d3","Type":"ContainerStarted","Data":"79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d"}
Oct 31 08:16:28.825799 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:28.825453    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:28.844454 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:28.844406    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podStartSLOduration=2.214571653 podStartE2EDuration="12.844393259s" totalImagesPullingTime="6.498286857s" totalInitContainerRuntime="4.131534749s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:16:28.842686132 +0000 UTC m=+2837.532454425" watchObservedRunningTime="2026-10-31 08:16:28.844393259 +0000 UTC m=+2837.534161550"
Oct 31 08:16:29.829679 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:29.829647    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:29.830987 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:29.830952    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:16:30.833702 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:30.833659    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:16:35.838104 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:35.838070    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:16:35.838568 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:35.838540    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:16:40.307341 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.307308    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"]
Oct 31 08:16:40.311205 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.311171    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.314710 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.314689    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Oct 31 08:16:40.314861 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.314846    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"storage-config\""
Oct 31 08:16:40.314938 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.314889    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Oct 31 08:16:40.315045 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.315026    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:16:40.315095 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.315031    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:16:40.315860 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.315840    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:16:40.315962 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.315902    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:16:40.323961 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.323935    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"]
Oct 31 08:16:40.452065 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.452028    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e620844d-887d-4a34-a6f6-fca6718c162b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.452244 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.452074    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.452244 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.452107    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e620844d-887d-4a34-a6f6-fca6718c162b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.452244 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.452218    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsm7q\" (UniqueName: \"kubernetes.io/projected/e620844d-887d-4a34-a6f6-fca6718c162b-kube-api-access-xsm7q\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.452351 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.452251    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.553366 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.553330    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.553504 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.553378    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e620844d-887d-4a34-a6f6-fca6718c162b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.553504 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.553415    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsm7q\" (UniqueName: \"kubernetes.io/projected/e620844d-887d-4a34-a6f6-fca6718c162b-kube-api-access-xsm7q\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.553504 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.553457    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.553628 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.553520    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e620844d-887d-4a34-a6f6-fca6718c162b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.553965 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.553928    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e620844d-887d-4a34-a6f6-fca6718c162b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.554109 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.554097    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.554276 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.554254    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.555844 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.555817    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e620844d-887d-4a34-a6f6-fca6718c162b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.561396 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.561342    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsm7q\" (UniqueName: \"kubernetes.io/projected/e620844d-887d-4a34-a6f6-fca6718c162b-kube-api-access-xsm7q\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.631662 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.631622    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:40.904782 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:40.904707    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"]
Oct 31 08:16:40.907847 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:16:40.907816    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode620844d_887d_4a34_a6f6_fca6718c162b.slice/crio-dbc66a7c074c8e889342295def8568f0abd0e6aba541c2d6d537ed7df9f43e74 WatchSource:0}: Error finding container dbc66a7c074c8e889342295def8568f0abd0e6aba541c2d6d537ed7df9f43e74: Status 404 returned error can't find the container with id dbc66a7c074c8e889342295def8568f0abd0e6aba541c2d6d537ed7df9f43e74
Oct 31 08:16:41.881383 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:41.881349    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" event={"ID":"e620844d-887d-4a34-a6f6-fca6718c162b","Type":"ContainerStarted","Data":"2ef9c1e26c13158b337e47f552ce975c80aaaf389f75971d4b857c60e85016e0"}
Oct 31 08:16:41.881756 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:41.881407    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" event={"ID":"e620844d-887d-4a34-a6f6-fca6718c162b","Type":"ContainerStarted","Data":"dbc66a7c074c8e889342295def8568f0abd0e6aba541c2d6d537ed7df9f43e74"}
Oct 31 08:16:42.887339 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:42.887304    4913 generic.go:356] "Generic (PLEG): container finished" podID="e620844d-887d-4a34-a6f6-fca6718c162b" containerID="2ef9c1e26c13158b337e47f552ce975c80aaaf389f75971d4b857c60e85016e0" exitCode=0
Oct 31 08:16:42.887721 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:42.887381    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" event={"ID":"e620844d-887d-4a34-a6f6-fca6718c162b","Type":"ContainerDied","Data":"2ef9c1e26c13158b337e47f552ce975c80aaaf389f75971d4b857c60e85016e0"}
Oct 31 08:16:43.893732 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:43.893699    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" event={"ID":"e620844d-887d-4a34-a6f6-fca6718c162b","Type":"ContainerStarted","Data":"7d4922ade83aa1a07fa8b94eddcad3b44e4f2196b768c3bbb1c328c82cc668de"}
Oct 31 08:16:43.894155 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:43.893757    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" event={"ID":"e620844d-887d-4a34-a6f6-fca6718c162b","Type":"ContainerStarted","Data":"3ca3bba09d7c2bc528d77ccc37190987a654b316e9e21a01dd0c16a106ce6a03"}
Oct 31 08:16:43.894155 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:43.893855    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:43.913199 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:43.913136    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podStartSLOduration=2.759695699 podStartE2EDuration="3.913123498s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.153427799s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:16:43.91174745 +0000 UTC m=+2852.601515758" watchObservedRunningTime="2026-10-31 08:16:43.913123498 +0000 UTC m=+2852.602891784"
Oct 31 08:16:44.898528 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:44.898493    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:44.900067 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:44.900037    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:16:45.839368 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:45.839326    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:16:45.903067 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:45.903023    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:16:49.399071 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:16:49.399033    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:16:50.907736 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:50.907701    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:16:50.908401 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:50.908372    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:16:55.838590 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:16:55.838549    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:17:00.909007 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:00.908959    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:17:05.839342 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:05.839299    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:17:10.909080 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:10.909034    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:17:15.839118 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:15.839078    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:17:19.399907 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:17:19.399872    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:17:20.909286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:20.909248    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:17:25.839177 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:25.839135    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:17:30.908791 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:30.908748    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:17:35.838760 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:35.838720    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:17:40.908332 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:40.908289    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:17:45.839465 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:45.839421    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:17:49.399764 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:17:49.399729    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:17:50.909221 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:50.909166    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:17:55.839137 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:17:55.839054    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:18:00.908805 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:00.908765    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:18:01.424498 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:01.424469    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:18:07.751756 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:07.751716    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"]
Oct 31 08:18:07.752272 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:07.752039    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kube-rbac-proxy" containerID="cri-o://144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9" gracePeriod=30
Oct 31 08:18:07.752272 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:07.752015    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" containerID="cri-o://79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d" gracePeriod=30
Oct 31 08:18:08.243498 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:08.243464    4913 generic.go:356] "Generic (PLEG): container finished" podID="4e755559-c934-42e0-8def-b8ee970d10d3" containerID="144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9" exitCode=0
Oct 31 08:18:08.243665 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:08.243526    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" event={"ID":"4e755559-c934-42e0-8def-b8ee970d10d3","Type":"ContainerDied","Data":"144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9"}
Oct 31 08:18:10.834832 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:10.834786    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.77:8643/healthz\": dial tcp 10.132.0.77:8643: connect: connection refused"
Oct 31 08:18:10.909353 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:10.909316    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:18:11.420236 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:11.420198    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.77:8080: connect: connection refused"
Oct 31 08:18:13.104194 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.104163    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:18:13.196536 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.196447    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e755559-c934-42e0-8def-b8ee970d10d3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e755559-c934-42e0-8def-b8ee970d10d3-kserve-provision-location\") pod \"4e755559-c934-42e0-8def-b8ee970d10d3\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") "
Oct 31 08:18:13.196536 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.196515    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-cabundle-cert\") pod \"4e755559-c934-42e0-8def-b8ee970d10d3\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") "
Oct 31 08:18:13.196760 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.196547    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e755559-c934-42e0-8def-b8ee970d10d3-kube-api-access-7f5n5\" (UniqueName: \"kubernetes.io/projected/4e755559-c934-42e0-8def-b8ee970d10d3-kube-api-access-7f5n5\") pod \"4e755559-c934-42e0-8def-b8ee970d10d3\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") "
Oct 31 08:18:13.196760 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.196572    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e755559-c934-42e0-8def-b8ee970d10d3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e755559-c934-42e0-8def-b8ee970d10d3-proxy-tls\") pod \"4e755559-c934-42e0-8def-b8ee970d10d3\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") "
Oct 31 08:18:13.196760 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.196592    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-kube-rbac-proxy-sar-config\") pod \"4e755559-c934-42e0-8def-b8ee970d10d3\" (UID: \"4e755559-c934-42e0-8def-b8ee970d10d3\") "
Oct 31 08:18:13.196991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.196957    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e755559-c934-42e0-8def-b8ee970d10d3-kserve-provision-location" pod "4e755559-c934-42e0-8def-b8ee970d10d3" (UID: "4e755559-c934-42e0-8def-b8ee970d10d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:18:13.196991 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.196980    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-kube-rbac-proxy-sar-config" pod "4e755559-c934-42e0-8def-b8ee970d10d3" (UID: "4e755559-c934-42e0-8def-b8ee970d10d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:18:13.197153 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.196969    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-cabundle-cert" pod "4e755559-c934-42e0-8def-b8ee970d10d3" (UID: "4e755559-c934-42e0-8def-b8ee970d10d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:18:13.198730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.198706    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e755559-c934-42e0-8def-b8ee970d10d3-kube-api-access-7f5n5" pod "4e755559-c934-42e0-8def-b8ee970d10d3" (UID: "4e755559-c934-42e0-8def-b8ee970d10d3"). InnerVolumeSpecName "kube-api-access-7f5n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:18:13.198809 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.198735    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e755559-c934-42e0-8def-b8ee970d10d3-proxy-tls" pod "4e755559-c934-42e0-8def-b8ee970d10d3" (UID: "4e755559-c934-42e0-8def-b8ee970d10d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:18:13.264282 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.264245    4913 generic.go:356] "Generic (PLEG): container finished" podID="4e755559-c934-42e0-8def-b8ee970d10d3" containerID="79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d" exitCode=0
Oct 31 08:18:13.264448 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.264300    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" event={"ID":"4e755559-c934-42e0-8def-b8ee970d10d3","Type":"ContainerDied","Data":"79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d"}
Oct 31 08:18:13.264448 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.264306    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"
Oct 31 08:18:13.264448 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.264351    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb" event={"ID":"4e755559-c934-42e0-8def-b8ee970d10d3","Type":"ContainerDied","Data":"9d085dcd3deff04db97d892b9bf34add57d286d9243b8ead74026dcd4f6ef421"}
Oct 31 08:18:13.264448 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.264377    4913 scope.go:122] "RemoveContainer" containerID="144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9"
Oct 31 08:18:13.273371 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.273356    4913 scope.go:122] "RemoveContainer" containerID="79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d"
Oct 31 08:18:13.280741 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.280724    4913 scope.go:122] "RemoveContainer" containerID="71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d"
Oct 31 08:18:13.284898 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.284876    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"]
Oct 31 08:18:13.288073 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.288057    4913 scope.go:122] "RemoveContainer" containerID="144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9"
Oct 31 08:18:13.288429 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:18:13.288402    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9\": container with ID starting with 144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9 not found: ID does not exist" containerID="144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9"
Oct 31 08:18:13.288482 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.288445    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9"} err="failed to get container status \"144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9\": rpc error: code = NotFound desc = could not find container \"144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9\": container with ID starting with 144cc354b8223d4b0cb484efa513f44fda9784fd2e1e5439681d30f585546ca9 not found: ID does not exist"
Oct 31 08:18:13.288482 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.288463    4913 scope.go:122] "RemoveContainer" containerID="79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d"
Oct 31 08:18:13.288700 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:18:13.288683    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d\": container with ID starting with 79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d not found: ID does not exist" containerID="79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d"
Oct 31 08:18:13.288744 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.288705    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d"} err="failed to get container status \"79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d\": rpc error: code = NotFound desc = could not find container \"79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d\": container with ID starting with 79fbda6258a0e76bf5da97721eee818405a5f0543f657c02e6a307a369f2a18d not found: ID does not exist"
Oct 31 08:18:13.288744 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.288718    4913 scope.go:122] "RemoveContainer" containerID="71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d"
Oct 31 08:18:13.288948 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:18:13.288926    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d\": container with ID starting with 71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d not found: ID does not exist" containerID="71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d"
Oct 31 08:18:13.289001 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.288953    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d"} err="failed to get container status \"71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d\": rpc error: code = NotFound desc = could not find container \"71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d\": container with ID starting with 71c7569c272e0beeb750b3f5a01eb835256764c9298fa26be6eef607dbfc515d not found: ID does not exist"
Oct 31 08:18:13.290702 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.290682    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-predictor-5b4d64fbc6-hrvtb"]
Oct 31 08:18:13.297838 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.297819    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e755559-c934-42e0-8def-b8ee970d10d3-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:13.297916 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.297840    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:13.297916 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.297853    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5n5\" (UniqueName: \"kubernetes.io/projected/4e755559-c934-42e0-8def-b8ee970d10d3-kube-api-access-7f5n5\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:13.297916 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.297862    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e755559-c934-42e0-8def-b8ee970d10d3-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:13.297916 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.297871    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e755559-c934-42e0-8def-b8ee970d10d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:13.424073 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:13.424041    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" path="/var/lib/kubelet/pods/4e755559-c934-42e0-8def-b8ee970d10d3/volumes"
Oct 31 08:18:15.236646 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.236611    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"]
Oct 31 08:18:15.237036 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.236935    4913 cpu_manager.go:414] "containerMap: removing container" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container"
Oct 31 08:18:15.237036 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.236947    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container"
Oct 31 08:18:15.237036 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.236962    4913 cpu_manager.go:414] "containerMap: removing container" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="storage-initializer"
Oct 31 08:18:15.237036 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.236967    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="storage-initializer"
Oct 31 08:18:15.237036 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.236976    4913 cpu_manager.go:414] "containerMap: removing container" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kube-rbac-proxy"
Oct 31 08:18:15.237036 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.236982    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kube-rbac-proxy"
Oct 31 08:18:15.237250 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.237053    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kube-rbac-proxy"
Oct 31 08:18:15.237250 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.237065    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e755559-c934-42e0-8def-b8ee970d10d3" containerName="kserve-container"
Oct 31 08:18:15.240444 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.240426    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.243677 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.243629    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Oct 31 08:18:15.243677 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.243642    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:18:15.243677 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.243642    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:18:15.244356 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.244340    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:18:15.244511 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.244493    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:18:15.244611 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.244569    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Oct 31 08:18:15.244611 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.244593    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:18:15.249088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.249060    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"]
Oct 31 08:18:15.313872 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.313836    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e824871e-e067-4d28-851b-18607cd679c8-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.314046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.313885    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e824871e-e067-4d28-851b-18607cd679c8-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.314046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.313993    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.314046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.314022    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7l4\" (UniqueName: \"kubernetes.io/projected/e824871e-e067-4d28-851b-18607cd679c8-kube-api-access-xr7l4\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.314046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.314043    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.414705 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.414664    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.414705 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.414702    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7l4\" (UniqueName: \"kubernetes.io/projected/e824871e-e067-4d28-851b-18607cd679c8-kube-api-access-xr7l4\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.414967 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.414719    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.414967 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.414748    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e824871e-e067-4d28-851b-18607cd679c8-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.414967 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.414915    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e824871e-e067-4d28-851b-18607cd679c8-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.415337 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.415316    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e824871e-e067-4d28-851b-18607cd679c8-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.415498 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.415478    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.415565 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.415483    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.417346 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.417322    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e824871e-e067-4d28-851b-18607cd679c8-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.422622 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.422591    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7l4\" (UniqueName: \"kubernetes.io/projected/e824871e-e067-4d28-851b-18607cd679c8-kube-api-access-xr7l4\") pod \"isvc-lightgbm-runtime-predictor-78d884686d-9t5cs\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.559567 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.559481    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:15.833367 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:15.833275    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"]
Oct 31 08:18:15.836550 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:18:15.836521    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode824871e_e067_4d28_851b_18607cd679c8.slice/crio-b4c46152a1e3dab530e59dc151b6dbc90dc9eeeeb0412f1fb99064227c95d319 WatchSource:0}: Error finding container b4c46152a1e3dab530e59dc151b6dbc90dc9eeeeb0412f1fb99064227c95d319: Status 404 returned error can't find the container with id b4c46152a1e3dab530e59dc151b6dbc90dc9eeeeb0412f1fb99064227c95d319
Oct 31 08:18:16.278445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:16.278407    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" event={"ID":"e824871e-e067-4d28-851b-18607cd679c8","Type":"ContainerStarted","Data":"189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0"}
Oct 31 08:18:16.278907 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:16.278469    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" event={"ID":"e824871e-e067-4d28-851b-18607cd679c8","Type":"ContainerStarted","Data":"b4c46152a1e3dab530e59dc151b6dbc90dc9eeeeb0412f1fb99064227c95d319"}
Oct 31 08:18:19.401045 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:18:19.400998    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:18:20.300144 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:20.300065    4913 generic.go:356] "Generic (PLEG): container finished" podID="e824871e-e067-4d28-851b-18607cd679c8" containerID="189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0" exitCode=0
Oct 31 08:18:20.300144 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:20.300134    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" event={"ID":"e824871e-e067-4d28-851b-18607cd679c8","Type":"ContainerDied","Data":"189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0"}
Oct 31 08:18:20.439762 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:20.439723    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"]
Oct 31 08:18:20.440212 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:20.440007    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" containerID="cri-o://3ca3bba09d7c2bc528d77ccc37190987a654b316e9e21a01dd0c16a106ce6a03" gracePeriod=30
Oct 31 08:18:20.440212 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:20.440044    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kube-rbac-proxy" containerID="cri-o://7d4922ade83aa1a07fa8b94eddcad3b44e4f2196b768c3bbb1c328c82cc668de" gracePeriod=30
Oct 31 08:18:20.903755 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:20.903667    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.78:8643/healthz\": dial tcp 10.132.0.78:8643: connect: connection refused"
Oct 31 08:18:20.908479 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:20.908451    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.78:8080: connect: connection refused"
Oct 31 08:18:21.306281 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:21.306243    4913 generic.go:356] "Generic (PLEG): container finished" podID="e620844d-887d-4a34-a6f6-fca6718c162b" containerID="7d4922ade83aa1a07fa8b94eddcad3b44e4f2196b768c3bbb1c328c82cc668de" exitCode=0
Oct 31 08:18:21.306481 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:21.306322    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" event={"ID":"e620844d-887d-4a34-a6f6-fca6718c162b","Type":"ContainerDied","Data":"7d4922ade83aa1a07fa8b94eddcad3b44e4f2196b768c3bbb1c328c82cc668de"}
Oct 31 08:18:21.308589 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:21.308561    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" event={"ID":"e824871e-e067-4d28-851b-18607cd679c8","Type":"ContainerStarted","Data":"df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1"}
Oct 31 08:18:21.308710 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:21.308624    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" event={"ID":"e824871e-e067-4d28-851b-18607cd679c8","Type":"ContainerStarted","Data":"d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6"}
Oct 31 08:18:21.308910 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:21.308893    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:22.312607 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:22.312571    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:22.313841 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:22.313816    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:18:23.317344 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:23.317307    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:18:25.328903 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.328868    4913 generic.go:356] "Generic (PLEG): container finished" podID="e620844d-887d-4a34-a6f6-fca6718c162b" containerID="3ca3bba09d7c2bc528d77ccc37190987a654b316e9e21a01dd0c16a106ce6a03" exitCode=0
Oct 31 08:18:25.329396 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.328933    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" event={"ID":"e620844d-887d-4a34-a6f6-fca6718c162b","Type":"ContainerDied","Data":"3ca3bba09d7c2bc528d77ccc37190987a654b316e9e21a01dd0c16a106ce6a03"}
Oct 31 08:18:25.384263 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.384236    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:18:25.402344 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.402304    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podStartSLOduration=6.323180455 podStartE2EDuration="10.402291797s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.079111342s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:18:21.327314837 +0000 UTC m=+2950.017083143" watchObservedRunningTime="2026-10-31 08:18:25.402291797 +0000 UTC m=+2954.092060091"
Oct 31 08:18:25.500047 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.499967    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e620844d-887d-4a34-a6f6-fca6718c162b-kube-api-access-xsm7q\" (UniqueName: \"kubernetes.io/projected/e620844d-887d-4a34-a6f6-fca6718c162b-kube-api-access-xsm7q\") pod \"e620844d-887d-4a34-a6f6-fca6718c162b\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") "
Oct 31 08:18:25.500047 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.500016    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e620844d-887d-4a34-a6f6-fca6718c162b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e620844d-887d-4a34-a6f6-fca6718c162b-kserve-provision-location\") pod \"e620844d-887d-4a34-a6f6-fca6718c162b\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") "
Oct 31 08:18:25.500047 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.500048    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-cabundle-cert\") pod \"e620844d-887d-4a34-a6f6-fca6718c162b\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") "
Oct 31 08:18:25.500360 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.500106    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-kube-rbac-proxy-sar-config\") pod \"e620844d-887d-4a34-a6f6-fca6718c162b\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") "
Oct 31 08:18:25.500360 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.500127    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e620844d-887d-4a34-a6f6-fca6718c162b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e620844d-887d-4a34-a6f6-fca6718c162b-proxy-tls\") pod \"e620844d-887d-4a34-a6f6-fca6718c162b\" (UID: \"e620844d-887d-4a34-a6f6-fca6718c162b\") "
Oct 31 08:18:25.500479 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.500420    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e620844d-887d-4a34-a6f6-fca6718c162b-kserve-provision-location" pod "e620844d-887d-4a34-a6f6-fca6718c162b" (UID: "e620844d-887d-4a34-a6f6-fca6718c162b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:18:25.500479 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.500471    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-cabundle-cert" pod "e620844d-887d-4a34-a6f6-fca6718c162b" (UID: "e620844d-887d-4a34-a6f6-fca6718c162b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:18:25.500586 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.500548    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-kube-rbac-proxy-sar-config" pod "e620844d-887d-4a34-a6f6-fca6718c162b" (UID: "e620844d-887d-4a34-a6f6-fca6718c162b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:18:25.502140 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.502114    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e620844d-887d-4a34-a6f6-fca6718c162b-kube-api-access-xsm7q" pod "e620844d-887d-4a34-a6f6-fca6718c162b" (UID: "e620844d-887d-4a34-a6f6-fca6718c162b"). InnerVolumeSpecName "kube-api-access-xsm7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:18:25.502269 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.502171    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e620844d-887d-4a34-a6f6-fca6718c162b-proxy-tls" pod "e620844d-887d-4a34-a6f6-fca6718c162b" (UID: "e620844d-887d-4a34-a6f6-fca6718c162b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:18:25.601167 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.601130    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsm7q\" (UniqueName: \"kubernetes.io/projected/e620844d-887d-4a34-a6f6-fca6718c162b-kube-api-access-xsm7q\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:25.601167 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.601159    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e620844d-887d-4a34-a6f6-fca6718c162b-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:25.601167 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.601170    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:25.601421 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.601180    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e620844d-887d-4a34-a6f6-fca6718c162b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:25.601421 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:25.601208    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e620844d-887d-4a34-a6f6-fca6718c162b-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:18:26.335410 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:26.335381    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"
Oct 31 08:18:26.335811 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:26.335384    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h" event={"ID":"e620844d-887d-4a34-a6f6-fca6718c162b","Type":"ContainerDied","Data":"dbc66a7c074c8e889342295def8568f0abd0e6aba541c2d6d537ed7df9f43e74"}
Oct 31 08:18:26.335811 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:26.335536    4913 scope.go:122] "RemoveContainer" containerID="7d4922ade83aa1a07fa8b94eddcad3b44e4f2196b768c3bbb1c328c82cc668de"
Oct 31 08:18:26.350051 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:26.350030    4913 scope.go:122] "RemoveContainer" containerID="3ca3bba09d7c2bc528d77ccc37190987a654b316e9e21a01dd0c16a106ce6a03"
Oct 31 08:18:26.357921 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:26.357905    4913 scope.go:122] "RemoveContainer" containerID="2ef9c1e26c13158b337e47f552ce975c80aaaf389f75971d4b857c60e85016e0"
Oct 31 08:18:26.359823 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:26.359799    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"]
Oct 31 08:18:26.363472 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:26.363446    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-global-pass-predictor-6b68bbf647-qj57h"]
Oct 31 08:18:27.419625 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:27.419591    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" path="/var/lib/kubelet/pods/e620844d-887d-4a34-a6f6-fca6718c162b/volumes"
Oct 31 08:18:28.321552 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:28.321517    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:18:28.322027 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:28.322001    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:18:33.044362 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044324    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"]
Oct 31 08:18:33.044750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044656    4913 cpu_manager.go:414] "containerMap: removing container" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kube-rbac-proxy"
Oct 31 08:18:33.044750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044667    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kube-rbac-proxy"
Oct 31 08:18:33.044750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044681    4913 cpu_manager.go:414] "containerMap: removing container" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container"
Oct 31 08:18:33.044750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044687    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container"
Oct 31 08:18:33.044750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044698    4913 cpu_manager.go:414] "containerMap: removing container" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="storage-initializer"
Oct 31 08:18:33.044750 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044703    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="storage-initializer"
Oct 31 08:18:33.044982 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044764    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kube-rbac-proxy"
Oct 31 08:18:33.044982 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.044774    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="e620844d-887d-4a34-a6f6-fca6718c162b" containerName="kserve-container"
Oct 31 08:18:33.049691 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.049669    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.057716 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.057683    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"storage-config\""
Oct 31 08:18:33.057857 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.057688    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Oct 31 08:18:33.057978 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.057753    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Oct 31 08:18:33.057978 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.057760    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:18:33.058159 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.057787    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:18:33.058159 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.057876    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:18:33.058159 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.058127    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:18:33.059559 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.059537    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"]
Oct 31 08:18:33.165074 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.165034    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/961deffd-b3ef-4a36-ab93-2cccff99fa4b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.165286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.165084    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.165286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.165105    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grht6\" (UniqueName: \"kubernetes.io/projected/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-api-access-grht6\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.165398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.165288    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.165398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.165348    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.266553 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.266515    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.266553 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.266555    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grht6\" (UniqueName: \"kubernetes.io/projected/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-api-access-grht6\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.266817 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.266580    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.266817 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.266614    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.266817 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.266649    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/961deffd-b3ef-4a36-ab93-2cccff99fa4b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.267028 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.267003    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.267464 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.267432    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.267565 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.267432    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.269087 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.269066    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/961deffd-b3ef-4a36-ab93-2cccff99fa4b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.273849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.273826    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grht6\" (UniqueName: \"kubernetes.io/projected/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-api-access-grht6\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.368897 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.368804    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:33.645347 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:33.645203    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"]
Oct 31 08:18:33.647904 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:18:33.647863    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961deffd_b3ef_4a36_ab93_2cccff99fa4b.slice/crio-bce59f8bbd484f9486c8530b0978a0d5903d10d4343f3697c1393cec60a0dceb WatchSource:0}: Error finding container bce59f8bbd484f9486c8530b0978a0d5903d10d4343f3697c1393cec60a0dceb: Status 404 returned error can't find the container with id bce59f8bbd484f9486c8530b0978a0d5903d10d4343f3697c1393cec60a0dceb
Oct 31 08:18:34.377460 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:34.377416    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" event={"ID":"961deffd-b3ef-4a36-ab93-2cccff99fa4b","Type":"ContainerStarted","Data":"ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217"}
Oct 31 08:18:34.377842 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:34.377499    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" event={"ID":"961deffd-b3ef-4a36-ab93-2cccff99fa4b","Type":"ContainerStarted","Data":"bce59f8bbd484f9486c8530b0978a0d5903d10d4343f3697c1393cec60a0dceb"}
Oct 31 08:18:35.383450 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:35.383415    4913 generic.go:356] "Generic (PLEG): container finished" podID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerID="ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217" exitCode=0
Oct 31 08:18:35.383881 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:35.383505    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" event={"ID":"961deffd-b3ef-4a36-ab93-2cccff99fa4b","Type":"ContainerDied","Data":"ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217"}
Oct 31 08:18:36.389764 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:36.389724    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" event={"ID":"961deffd-b3ef-4a36-ab93-2cccff99fa4b","Type":"ContainerStarted","Data":"55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a"}
Oct 31 08:18:36.390210 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:36.389798    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" event={"ID":"961deffd-b3ef-4a36-ab93-2cccff99fa4b","Type":"ContainerStarted","Data":"d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7"}
Oct 31 08:18:36.390210 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:36.389833    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:36.408034 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:36.407984    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podStartSLOduration=2.236118656 podStartE2EDuration="3.407966981s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.171848325s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:18:36.406687891 +0000 UTC m=+2965.096456209" watchObservedRunningTime="2026-10-31 08:18:36.407966981 +0000 UTC m=+2965.097735274"
Oct 31 08:18:37.395088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:37.395057    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:37.396470 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:37.396443    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:18:38.321995 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:38.321950    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:18:38.402504 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:38.402463    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:18:43.407000 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:43.406969    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:18:43.407439 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:43.407305    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:18:48.322479 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:48.322435    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:18:49.399574 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:18:49.399538    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:18:53.407451 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:53.407410    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:18:58.322046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:18:58.322000    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:19:03.408098 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:03.408056    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:19:08.322828 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:08.322787    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:19:11.686305 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:11.686273    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:19:11.692088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:11.692064    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:19:13.407450 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:13.407411    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:19:18.321979 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:18.321930    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:19:19.400028 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:19:19.399991    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:19:23.407793 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:23.407708    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:19:28.321985 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:28.321945    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:19:33.408326 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:33.408284    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:19:38.322535 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:38.322491    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:19:39.415016 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:39.414964    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:19:43.407359 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:43.407316    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:19:49.399317 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:19:49.399273    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:19:49.415643 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:49.415613    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:19:53.407674 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:53.407629    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:19:59.419378 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:19:59.419352    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:20:03.408433 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:03.408393    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:20:06.580924 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:06.580885    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"]
Oct 31 08:20:06.581420 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:06.581179    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" containerID="cri-o://d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6" gracePeriod=30
Oct 31 08:20:06.581420 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:06.581363    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kube-rbac-proxy" containerID="cri-o://df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1" gracePeriod=30
Oct 31 08:20:06.766225 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:06.766170    4913 generic.go:356] "Generic (PLEG): container finished" podID="e824871e-e067-4d28-851b-18607cd679c8" containerID="df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1" exitCode=0
Oct 31 08:20:06.766394 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:06.766229    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" event={"ID":"e824871e-e067-4d28-851b-18607cd679c8","Type":"ContainerDied","Data":"df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1"}
Oct 31 08:20:08.317993 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:08.317954    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.79:8643/healthz\": dial tcp 10.132.0.79:8643: connect: connection refused"
Oct 31 08:20:09.414912 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:09.414867    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.79:8080: connect: connection refused"
Oct 31 08:20:12.320031 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.320003    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:20:12.408667 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.408629    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e824871e-e067-4d28-851b-18607cd679c8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e824871e-e067-4d28-851b-18607cd679c8-kserve-provision-location\") pod \"e824871e-e067-4d28-851b-18607cd679c8\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") "
Oct 31 08:20:12.408863 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.408687    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-kube-rbac-proxy-sar-config\") pod \"e824871e-e067-4d28-851b-18607cd679c8\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") "
Oct 31 08:20:12.408863 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.408737    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-cabundle-cert\") pod \"e824871e-e067-4d28-851b-18607cd679c8\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") "
Oct 31 08:20:12.408863 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.408768    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e824871e-e067-4d28-851b-18607cd679c8-kube-api-access-xr7l4\" (UniqueName: \"kubernetes.io/projected/e824871e-e067-4d28-851b-18607cd679c8-kube-api-access-xr7l4\") pod \"e824871e-e067-4d28-851b-18607cd679c8\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") "
Oct 31 08:20:12.408863 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.408844    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e824871e-e067-4d28-851b-18607cd679c8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e824871e-e067-4d28-851b-18607cd679c8-proxy-tls\") pod \"e824871e-e067-4d28-851b-18607cd679c8\" (UID: \"e824871e-e067-4d28-851b-18607cd679c8\") "
Oct 31 08:20:12.409075 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.409016    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e824871e-e067-4d28-851b-18607cd679c8-kserve-provision-location" pod "e824871e-e067-4d28-851b-18607cd679c8" (UID: "e824871e-e067-4d28-851b-18607cd679c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:20:12.409075 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.409038    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-kube-rbac-proxy-sar-config" pod "e824871e-e067-4d28-851b-18607cd679c8" (UID: "e824871e-e067-4d28-851b-18607cd679c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:20:12.409144 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.409121    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-cabundle-cert" pod "e824871e-e067-4d28-851b-18607cd679c8" (UID: "e824871e-e067-4d28-851b-18607cd679c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:20:12.410993 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.410960    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e824871e-e067-4d28-851b-18607cd679c8-kube-api-access-xr7l4" pod "e824871e-e067-4d28-851b-18607cd679c8" (UID: "e824871e-e067-4d28-851b-18607cd679c8"). InnerVolumeSpecName "kube-api-access-xr7l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:20:12.410993 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.410976    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e824871e-e067-4d28-851b-18607cd679c8-proxy-tls" pod "e824871e-e067-4d28-851b-18607cd679c8" (UID: "e824871e-e067-4d28-851b-18607cd679c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:20:12.510436 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.510338    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:12.510436 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.510377    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr7l4\" (UniqueName: \"kubernetes.io/projected/e824871e-e067-4d28-851b-18607cd679c8-kube-api-access-xr7l4\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:12.510436 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.510393    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e824871e-e067-4d28-851b-18607cd679c8-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:12.510436 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.510405    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e824871e-e067-4d28-851b-18607cd679c8-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:12.510436 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.510418    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e824871e-e067-4d28-851b-18607cd679c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:12.793866 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.793786    4913 generic.go:356] "Generic (PLEG): container finished" podID="e824871e-e067-4d28-851b-18607cd679c8" containerID="d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6" exitCode=0
Oct 31 08:20:12.793866 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.793853    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"
Oct 31 08:20:12.794053 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.793869    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" event={"ID":"e824871e-e067-4d28-851b-18607cd679c8","Type":"ContainerDied","Data":"d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6"}
Oct 31 08:20:12.794053 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.793930    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs" event={"ID":"e824871e-e067-4d28-851b-18607cd679c8","Type":"ContainerDied","Data":"b4c46152a1e3dab530e59dc151b6dbc90dc9eeeeb0412f1fb99064227c95d319"}
Oct 31 08:20:12.794053 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.793959    4913 scope.go:122] "RemoveContainer" containerID="df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1"
Oct 31 08:20:12.808176 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.808154    4913 scope.go:122] "RemoveContainer" containerID="d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6"
Oct 31 08:20:12.815563 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.815547    4913 scope.go:122] "RemoveContainer" containerID="189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0"
Oct 31 08:20:12.819521 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.819495    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"]
Oct 31 08:20:12.822134 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.822112    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-runtime-predictor-78d884686d-9t5cs"]
Oct 31 08:20:12.824129 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.824108    4913 scope.go:122] "RemoveContainer" containerID="df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1"
Oct 31 08:20:12.824400 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:12.824380    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1\": container with ID starting with df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1 not found: ID does not exist" containerID="df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1"
Oct 31 08:20:12.824482 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.824413    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1"} err="failed to get container status \"df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1\": rpc error: code = NotFound desc = could not find container \"df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1\": container with ID starting with df41005a92eb9573392c20dafdeceecb13d4d27a15182188b48b10127d6055d1 not found: ID does not exist"
Oct 31 08:20:12.824482 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.824439    4913 scope.go:122] "RemoveContainer" containerID="d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6"
Oct 31 08:20:12.824694 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:12.824676    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6\": container with ID starting with d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6 not found: ID does not exist" containerID="d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6"
Oct 31 08:20:12.824736 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.824701    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6"} err="failed to get container status \"d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6\": rpc error: code = NotFound desc = could not find container \"d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6\": container with ID starting with d18b53d1fdb09a191db16edb0d14734991ce5dabf7be398f3090f1c3ac877bc6 not found: ID does not exist"
Oct 31 08:20:12.824736 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.824717    4913 scope.go:122] "RemoveContainer" containerID="189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0"
Oct 31 08:20:12.824930 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:12.824912    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0\": container with ID starting with 189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0 not found: ID does not exist" containerID="189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0"
Oct 31 08:20:12.824982 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:12.824938    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0"} err="failed to get container status \"189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0\": rpc error: code = NotFound desc = could not find container \"189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0\": container with ID starting with 189d48cb1fe9bcf3fee70a7b7c241601f58ddb287ba2a0e6a2c7b230b59e94b0 not found: ID does not exist"
Oct 31 08:20:13.174410 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:13.174375    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"]
Oct 31 08:20:13.174726 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:13.174693    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" containerID="cri-o://d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7" gracePeriod=30
Oct 31 08:20:13.174819 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:13.174709    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kube-rbac-proxy" containerID="cri-o://55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a" gracePeriod=30
Oct 31 08:20:13.403194 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:13.403144    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.80:8643/healthz\": dial tcp 10.132.0.80:8643: connect: connection refused"
Oct 31 08:20:13.407383 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:13.407362    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.80:8080: connect: connection refused"
Oct 31 08:20:13.418694 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:13.418666    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e824871e-e067-4d28-851b-18607cd679c8" path="/var/lib/kubelet/pods/e824871e-e067-4d28-851b-18607cd679c8/volumes"
Oct 31 08:20:13.799736 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:13.799704    4913 generic.go:356] "Generic (PLEG): container finished" podID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerID="55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a" exitCode=0
Oct 31 08:20:13.799921 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:13.799781    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" event={"ID":"961deffd-b3ef-4a36-ab93-2cccff99fa4b","Type":"ContainerDied","Data":"55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a"}
Oct 31 08:20:14.412862 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.412829    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"]
Oct 31 08:20:14.413287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.413157    4913 cpu_manager.go:414] "containerMap: removing container" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container"
Oct 31 08:20:14.413287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.413169    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container"
Oct 31 08:20:14.413287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.413177    4913 cpu_manager.go:414] "containerMap: removing container" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kube-rbac-proxy"
Oct 31 08:20:14.413287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.413196    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kube-rbac-proxy"
Oct 31 08:20:14.413287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.413212    4913 cpu_manager.go:414] "containerMap: removing container" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="storage-initializer"
Oct 31 08:20:14.413287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.413217    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="storage-initializer"
Oct 31 08:20:14.413287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.413275    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kserve-container"
Oct 31 08:20:14.413287 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.413284    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="e824871e-e067-4d28-851b-18607cd679c8" containerName="kube-rbac-proxy"
Oct 31 08:20:14.416909 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.416885    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.420254 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.420220    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Oct 31 08:20:14.421328 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.421307    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Oct 31 08:20:14.425637 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.425616    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"]
Oct 31 08:20:14.528521 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.528489    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.528665 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.528532    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.528769 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.528756    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.528897 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.528877    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qs8\" (UniqueName: \"kubernetes.io/projected/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-api-access-k9qs8\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.629451 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.629408    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.629638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.629462    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qs8\" (UniqueName: \"kubernetes.io/projected/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-api-access-k9qs8\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.629638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.629491    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.629638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.629514    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.629923 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.629900    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.630257 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.630235    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.631909 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.631889    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.639737 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.639711    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qs8\" (UniqueName: \"kubernetes.io/projected/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-api-access-k9qs8\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:14.737037 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:14.736996    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:15.011736 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:15.011709    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"]
Oct 31 08:20:15.014338 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:20:15.014308    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e00f46_a5f7_4ab7_95bb_12de9283c1cd.slice/crio-62ee5f96e900b9b4e172adfce1afa1693db7c716a8fbffdca15b821e8aae42e1 WatchSource:0}: Error finding container 62ee5f96e900b9b4e172adfce1afa1693db7c716a8fbffdca15b821e8aae42e1: Status 404 returned error can't find the container with id 62ee5f96e900b9b4e172adfce1afa1693db7c716a8fbffdca15b821e8aae42e1
Oct 31 08:20:15.812098 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:15.812058    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k" event={"ID":"16e00f46-a5f7-4ab7-95bb-12de9283c1cd","Type":"ContainerStarted","Data":"fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7"}
Oct 31 08:20:15.812608 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:15.812130    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k" event={"ID":"16e00f46-a5f7-4ab7-95bb-12de9283c1cd","Type":"ContainerStarted","Data":"62ee5f96e900b9b4e172adfce1afa1693db7c716a8fbffdca15b821e8aae42e1"}
Oct 31 08:20:18.420574 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.420548    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:20:18.463069 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.463037    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-api-access-grht6\" (UniqueName: \"kubernetes.io/projected/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-api-access-grht6\") pod \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") "
Oct 31 08:20:18.463230 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.463131    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/961deffd-b3ef-4a36-ab93-2cccff99fa4b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/961deffd-b3ef-4a36-ab93-2cccff99fa4b-proxy-tls\") pod \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") "
Oct 31 08:20:18.463230 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.463171    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-rbac-proxy-sar-config\") pod \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") "
Oct 31 08:20:18.463338 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.463254    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-cabundle-cert\") pod \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") "
Oct 31 08:20:18.463338 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.463290    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kserve-provision-location\") pod \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\" (UID: \"961deffd-b3ef-4a36-ab93-2cccff99fa4b\") "
Oct 31 08:20:18.463612 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.463584    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-rbac-proxy-sar-config" pod "961deffd-b3ef-4a36-ab93-2cccff99fa4b" (UID: "961deffd-b3ef-4a36-ab93-2cccff99fa4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:20:18.463733 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.463632    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-cabundle-cert" pod "961deffd-b3ef-4a36-ab93-2cccff99fa4b" (UID: "961deffd-b3ef-4a36-ab93-2cccff99fa4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:20:18.463733 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.463640    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kserve-provision-location" pod "961deffd-b3ef-4a36-ab93-2cccff99fa4b" (UID: "961deffd-b3ef-4a36-ab93-2cccff99fa4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:20:18.465387 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.465366    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961deffd-b3ef-4a36-ab93-2cccff99fa4b-proxy-tls" pod "961deffd-b3ef-4a36-ab93-2cccff99fa4b" (UID: "961deffd-b3ef-4a36-ab93-2cccff99fa4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:20:18.465568 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.465547    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-api-access-grht6" pod "961deffd-b3ef-4a36-ab93-2cccff99fa4b" (UID: "961deffd-b3ef-4a36-ab93-2cccff99fa4b"). InnerVolumeSpecName "kube-api-access-grht6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:20:18.564089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.564007    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/961deffd-b3ef-4a36-ab93-2cccff99fa4b-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:18.564089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.564039    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:18.564089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.564049    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/961deffd-b3ef-4a36-ab93-2cccff99fa4b-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:18.564089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.564059    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:18.564089 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.564068    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grht6\" (UniqueName: \"kubernetes.io/projected/961deffd-b3ef-4a36-ab93-2cccff99fa4b-kube-api-access-grht6\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:18.826497 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.826402    4913 generic.go:356] "Generic (PLEG): container finished" podID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerID="d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7" exitCode=0
Oct 31 08:20:18.826497 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.826438    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" event={"ID":"961deffd-b3ef-4a36-ab93-2cccff99fa4b","Type":"ContainerDied","Data":"d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7"}
Oct 31 08:20:18.826497 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.826470    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"
Oct 31 08:20:18.826497 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.826485    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" event={"ID":"961deffd-b3ef-4a36-ab93-2cccff99fa4b","Type":"ContainerDied","Data":"bce59f8bbd484f9486c8530b0978a0d5903d10d4343f3697c1393cec60a0dceb"}
Oct 31 08:20:18.826807 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.826511    4913 scope.go:122] "RemoveContainer" containerID="55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a"
Oct 31 08:20:18.835124 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.835106    4913 scope.go:122] "RemoveContainer" containerID="d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7"
Oct 31 08:20:18.842540 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.842521    4913 scope.go:122] "RemoveContainer" containerID="ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217"
Oct 31 08:20:18.849102 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.849075    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"]
Oct 31 08:20:18.850776 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.850755    4913 scope.go:122] "RemoveContainer" containerID="55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a"
Oct 31 08:20:18.851126 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:18.851095    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a\": container with ID starting with 55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a not found: ID does not exist" containerID="55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a"
Oct 31 08:20:18.851237 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.851136    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a"} err="failed to get container status \"55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a\": rpc error: code = NotFound desc = could not find container \"55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a\": container with ID starting with 55b009d313a3158c9fccb0ead54f9834f337c4f8035417abc67401880dff009a not found: ID does not exist"
Oct 31 08:20:18.851237 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.851162    4913 scope.go:122] "RemoveContainer" containerID="d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7"
Oct 31 08:20:18.854586 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.854548    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk"]
Oct 31 08:20:18.856579 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:18.856549    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7\": container with ID starting with d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7 not found: ID does not exist" containerID="d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7"
Oct 31 08:20:18.856671 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.856588    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7"} err="failed to get container status \"d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7\": rpc error: code = NotFound desc = could not find container \"d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7\": container with ID starting with d3235d32b0234d081fa31e3770faa70c9f8a27cbf3d3d079d61d67725a0a97d7 not found: ID does not exist"
Oct 31 08:20:18.856671 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.856611    4913 scope.go:122] "RemoveContainer" containerID="ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217"
Oct 31 08:20:18.856927 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:18.856896    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217\": container with ID starting with ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217 not found: ID does not exist" containerID="ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217"
Oct 31 08:20:18.857016 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:18.856934    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217"} err="failed to get container status \"ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217\": rpc error: code = NotFound desc = could not find container \"ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217\": container with ID starting with ed9e5b2d006996ee3f094d07e5085d53b9489b2b42d5428223a1e065b0899217 not found: ID does not exist"
Oct 31 08:20:19.399220 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:19.399163    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:20:19.403408 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:19.403370    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-pass-predictor-7fbd9bc47-tlrzk" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.80:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:20:19.419753 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:19.419730    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" path="/var/lib/kubelet/pods/961deffd-b3ef-4a36-ab93-2cccff99fa4b/volumes"
Oct 31 08:20:19.833353 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:19.833323    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k_16e00f46-a5f7-4ab7-95bb-12de9283c1cd/storage-initializer/0.log"
Oct 31 08:20:19.833739 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:19.833358    4913 generic.go:356] "Generic (PLEG): container finished" podID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerID="fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7" exitCode=1
Oct 31 08:20:19.833739 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:19.833434    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k" event={"ID":"16e00f46-a5f7-4ab7-95bb-12de9283c1cd","Type":"ContainerDied","Data":"fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7"}
Oct 31 08:20:20.838734 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:20.838703    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k_16e00f46-a5f7-4ab7-95bb-12de9283c1cd/storage-initializer/0.log"
Oct 31 08:20:20.839149 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:20.838821    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k" event={"ID":"16e00f46-a5f7-4ab7-95bb-12de9283c1cd","Type":"ContainerStarted","Data":"28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4"}
Oct 31 08:20:24.377325 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.377287    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"]
Oct 31 08:20:24.377711 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.377555    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k" podUID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerName="storage-initializer" containerID="cri-o://28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4" gracePeriod=30
Oct 31 08:20:24.520913 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.520890    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k_16e00f46-a5f7-4ab7-95bb-12de9283c1cd/storage-initializer/1.log"
Oct 31 08:20:24.521295 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.521275    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k_16e00f46-a5f7-4ab7-95bb-12de9283c1cd/storage-initializer/0.log"
Oct 31 08:20:24.523672 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.523652    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:24.611469 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.611432    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-rbac-proxy-sar-config\") pod \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") "
Oct 31 08:20:24.611634 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.611490    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-proxy-tls\") pod \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") "
Oct 31 08:20:24.611634 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.611541    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-api-access-k9qs8\" (UniqueName: \"kubernetes.io/projected/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-api-access-k9qs8\") pod \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") "
Oct 31 08:20:24.611634 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.611577    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kserve-provision-location\") pod \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\" (UID: \"16e00f46-a5f7-4ab7-95bb-12de9283c1cd\") "
Oct 31 08:20:24.611874 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.611846    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kserve-provision-location" pod "16e00f46-a5f7-4ab7-95bb-12de9283c1cd" (UID: "16e00f46-a5f7-4ab7-95bb-12de9283c1cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:20:24.611915 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.611881    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-rbac-proxy-sar-config" pod "16e00f46-a5f7-4ab7-95bb-12de9283c1cd" (UID: "16e00f46-a5f7-4ab7-95bb-12de9283c1cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:20:24.613653 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.613625    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-proxy-tls" pod "16e00f46-a5f7-4ab7-95bb-12de9283c1cd" (UID: "16e00f46-a5f7-4ab7-95bb-12de9283c1cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:20:24.613765 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.613674    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-api-access-k9qs8" pod "16e00f46-a5f7-4ab7-95bb-12de9283c1cd" (UID: "16e00f46-a5f7-4ab7-95bb-12de9283c1cd"). InnerVolumeSpecName "kube-api-access-k9qs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:20:24.712307 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.712273    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9qs8\" (UniqueName: \"kubernetes.io/projected/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-api-access-k9qs8\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:24.712307 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.712303    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:24.712307 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.712313    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:24.712539 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.712324    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e00f46-a5f7-4ab7-95bb-12de9283c1cd-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:20:24.862697 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.857788    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k_16e00f46-a5f7-4ab7-95bb-12de9283c1cd/storage-initializer/1.log"
Oct 31 08:20:24.863273 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.863251    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k_16e00f46-a5f7-4ab7-95bb-12de9283c1cd/storage-initializer/0.log"
Oct 31 08:20:24.863360 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.863298    4913 generic.go:356] "Generic (PLEG): container finished" podID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerID="28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4" exitCode=1
Oct 31 08:20:24.863406 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.863379    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"
Oct 31 08:20:24.863406 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.863384    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k" event={"ID":"16e00f46-a5f7-4ab7-95bb-12de9283c1cd","Type":"ContainerDied","Data":"28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4"}
Oct 31 08:20:24.863480 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.863473    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k" event={"ID":"16e00f46-a5f7-4ab7-95bb-12de9283c1cd","Type":"ContainerDied","Data":"62ee5f96e900b9b4e172adfce1afa1693db7c716a8fbffdca15b821e8aae42e1"}
Oct 31 08:20:24.863513 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.863500    4913 scope.go:122] "RemoveContainer" containerID="28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4"
Oct 31 08:20:24.872432 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.872402    4913 scope.go:122] "RemoveContainer" containerID="fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7"
Oct 31 08:20:24.880041 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.880022    4913 scope.go:122] "RemoveContainer" containerID="28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4"
Oct 31 08:20:24.880315 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:24.880297    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4\": container with ID starting with 28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4 not found: ID does not exist" containerID="28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4"
Oct 31 08:20:24.880382 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.880321    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4"} err="failed to get container status \"28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4\": rpc error: code = NotFound desc = could not find container \"28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4\": container with ID starting with 28c3a9759199309e58cbd92d1ca2abc581e06dfb5b8be205bc90bd7d565ce0f4 not found: ID does not exist"
Oct 31 08:20:24.880382 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.880337    4913 scope.go:122] "RemoveContainer" containerID="fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7"
Oct 31 08:20:24.880544 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:24.880529    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7\": container with ID starting with fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7 not found: ID does not exist" containerID="fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7"
Oct 31 08:20:24.880580 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.880549    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7"} err="failed to get container status \"fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7\": rpc error: code = NotFound desc = could not find container \"fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7\": container with ID starting with fc3f5268a0c5b0eb49954608221f4ca665bf6c6989492062b8cc691b7a5cfeb7 not found: ID does not exist"
Oct 31 08:20:24.898728 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.898705    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"]
Oct 31 08:20:24.901967 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:24.901942    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-custom-fail-predictor-575bd849f9-qnh4k"]
Oct 31 08:20:25.419067 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.419026    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" path="/var/lib/kubelet/pods/16e00f46-a5f7-4ab7-95bb-12de9283c1cd/volumes"
Oct 31 08:20:25.746236 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746200    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"]
Oct 31 08:20:25.746523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746511    4913 cpu_manager.go:414] "containerMap: removing container" podUID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerName="storage-initializer"
Oct 31 08:20:25.746575 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746524    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerName="storage-initializer"
Oct 31 08:20:25.746575 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746534    4913 cpu_manager.go:414] "containerMap: removing container" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="storage-initializer"
Oct 31 08:20:25.746575 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746540    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="storage-initializer"
Oct 31 08:20:25.746575 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746550    4913 cpu_manager.go:414] "containerMap: removing container" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kube-rbac-proxy"
Oct 31 08:20:25.746575 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746556    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kube-rbac-proxy"
Oct 31 08:20:25.746575 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746570    4913 cpu_manager.go:414] "containerMap: removing container" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container"
Oct 31 08:20:25.746575 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746575    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container"
Oct 31 08:20:25.746795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746629    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerName="storage-initializer"
Oct 31 08:20:25.746795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746637    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kube-rbac-proxy"
Oct 31 08:20:25.746795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746646    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="961deffd-b3ef-4a36-ab93-2cccff99fa4b" containerName="kserve-container"
Oct 31 08:20:25.746795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746653    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerName="storage-initializer"
Oct 31 08:20:25.746795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746710    4913 cpu_manager.go:414] "containerMap: removing container" podUID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerName="storage-initializer"
Oct 31 08:20:25.746795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.746714    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e00f46-a5f7-4ab7-95bb-12de9283c1cd" containerName="storage-initializer"
Oct 31 08:20:25.751123 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.751106    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.756949 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.756909    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Oct 31 08:20:25.758732 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.758711    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"]
Oct 31 08:20:25.759795 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.759768    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:20:25.759888 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.759798    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:20:25.759888 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.759820    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:20:25.760082 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.760065    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"storage-config\""
Oct 31 08:20:25.760140 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.760123    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Oct 31 08:20:25.760290 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.760270    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:20:25.821906 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.821870    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.822081 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.821914    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6914600e-79c0-4f12-8e30-f2f531feaa44-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.822081 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.821944    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6914600e-79c0-4f12-8e30-f2f531feaa44-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.822081 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.822071    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxf8\" (UniqueName: \"kubernetes.io/projected/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-api-access-lsxf8\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.822211 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.822098    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.923158 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.923127    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6914600e-79c0-4f12-8e30-f2f531feaa44-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.923158 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.923164    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6914600e-79c0-4f12-8e30-f2f531feaa44-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.923388 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.923206    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxf8\" (UniqueName: \"kubernetes.io/projected/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-api-access-lsxf8\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.923388 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.923235    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.923388 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.923331    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.923638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.923613    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6914600e-79c0-4f12-8e30-f2f531feaa44-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.923913 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.923892    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.924003 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.923955    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.925716 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.925695    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6914600e-79c0-4f12-8e30-f2f531feaa44-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:25.932947 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:25.932926    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxf8\" (UniqueName: \"kubernetes.io/projected/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-api-access-lsxf8\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:26.069174 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:26.069079    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:26.342533 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:26.342454    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"]
Oct 31 08:20:26.345627 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:20:26.345593    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6914600e_79c0_4f12_8e30_f2f531feaa44.slice/crio-f050384b64dd98160fc437783dda37d32656246c862be52878ec113e2bf3a440 WatchSource:0}: Error finding container f050384b64dd98160fc437783dda37d32656246c862be52878ec113e2bf3a440: Status 404 returned error can't find the container with id f050384b64dd98160fc437783dda37d32656246c862be52878ec113e2bf3a440
Oct 31 08:20:26.873844 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:26.873804    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" event={"ID":"6914600e-79c0-4f12-8e30-f2f531feaa44","Type":"ContainerStarted","Data":"731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1"}
Oct 31 08:20:26.874251 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:26.873874    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" event={"ID":"6914600e-79c0-4f12-8e30-f2f531feaa44","Type":"ContainerStarted","Data":"f050384b64dd98160fc437783dda37d32656246c862be52878ec113e2bf3a440"}
Oct 31 08:20:27.880678 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:27.880599    4913 generic.go:356] "Generic (PLEG): container finished" podID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerID="731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1" exitCode=0
Oct 31 08:20:27.880678 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:27.880648    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" event={"ID":"6914600e-79c0-4f12-8e30-f2f531feaa44","Type":"ContainerDied","Data":"731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1"}
Oct 31 08:20:28.887892 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:28.887857    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" event={"ID":"6914600e-79c0-4f12-8e30-f2f531feaa44","Type":"ContainerStarted","Data":"253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743"}
Oct 31 08:20:28.888364 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:28.887923    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" event={"ID":"6914600e-79c0-4f12-8e30-f2f531feaa44","Type":"ContainerStarted","Data":"959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3"}
Oct 31 08:20:28.888364 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:28.888006    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:28.906893 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:28.906843    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podStartSLOduration=2.775024729 podStartE2EDuration="3.906832017s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.131807288s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:20:28.905731515 +0000 UTC m=+3077.595499809" watchObservedRunningTime="2026-10-31 08:20:28.906832017 +0000 UTC m=+3077.596600309"
Oct 31 08:20:29.892398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:29.892363    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:29.893925 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:29.893897    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:20:30.902292 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:30.902247    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:20:35.903368 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:35.903336    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:20:35.903819 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:35.903793    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:20:45.904244 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:45.904200    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:20:49.399336 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:20:49.399300    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:20:55.904377 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:20:55.904292    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:21:05.904151 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:05.904108    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:21:07.734374 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.734334    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"]
Oct 31 08:21:07.738005 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.737984    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.741657 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.741633    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"kube-root-ca.crt\""
Oct 31 08:21:07.747467 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.747445    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Oct 31 08:21:07.747673 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.747563    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"seaweedfs-s3-creds\""
Oct 31 08:21:07.747808 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.747754    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-13001498cc\"/\"default-dockercfg-4b4kb\""
Oct 31 08:21:07.749525 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.749501    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"]
Oct 31 08:21:07.763402 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.763373    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.763519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.763417    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.763519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.763498    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.763604 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.763543    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtc8\" (UniqueName: \"kubernetes.io/projected/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-api-access-zvtc8\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.763676 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.763658    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.842019 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.841978    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:21:07.842019 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.842016    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"openshift-service-ca.crt\""
Oct 31 08:21:07.842234 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.842096    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-13001498cc\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 31 08:21:07.864727 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.864680    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.864824 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.864756    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.864824 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.864779    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.864941 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.864916    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.865023 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.865008    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtc8\" (UniqueName: \"kubernetes.io/projected/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-api-access-zvtc8\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.865157 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.865132    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.865510 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.865493    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.865585 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.865551    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.867334 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.867312    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:07.872436 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:07.872411    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtc8\" (UniqueName: \"kubernetes.io/projected/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-api-access-zvtc8\") pod \"isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") " pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:08.055786 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:08.055700    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:08.330262 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:08.330210    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"]
Oct 31 08:21:08.332523 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:21:08.332491    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad138610_d7a2_4d79_9f04_2b1cd7f20b46.slice/crio-a11a1c983d1a1ff4a2797df93f58ec1ff9adc6f411f7f00e46c66b3be591c193 WatchSource:0}: Error finding container a11a1c983d1a1ff4a2797df93f58ec1ff9adc6f411f7f00e46c66b3be591c193: Status 404 returned error can't find the container with id a11a1c983d1a1ff4a2797df93f58ec1ff9adc6f411f7f00e46c66b3be591c193
Oct 31 08:21:09.071830 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:09.071791    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" event={"ID":"ad138610-d7a2-4d79-9f04-2b1cd7f20b46","Type":"ContainerStarted","Data":"1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa"}
Oct 31 08:21:09.072262 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:09.071911    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" event={"ID":"ad138610-d7a2-4d79-9f04-2b1cd7f20b46","Type":"ContainerStarted","Data":"a11a1c983d1a1ff4a2797df93f58ec1ff9adc6f411f7f00e46c66b3be591c193"}
Oct 31 08:21:13.090842 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:13.090800    4913 generic.go:356] "Generic (PLEG): container finished" podID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerID="1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa" exitCode=0
Oct 31 08:21:13.091246 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:13.090857    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" event={"ID":"ad138610-d7a2-4d79-9f04-2b1cd7f20b46","Type":"ContainerDied","Data":"1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa"}
Oct 31 08:21:14.101648 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:14.101614    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" event={"ID":"ad138610-d7a2-4d79-9f04-2b1cd7f20b46","Type":"ContainerStarted","Data":"7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d"}
Oct 31 08:21:14.102037 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:14.101676    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" event={"ID":"ad138610-d7a2-4d79-9f04-2b1cd7f20b46","Type":"ContainerStarted","Data":"ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f"}
Oct 31 08:21:14.102037 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:14.101998    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:14.102166 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:14.102145    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:14.103587 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:14.103554    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 31 08:21:14.122047 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:14.121997    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" podStartSLOduration=2.965078761 podStartE2EDuration="7.121984009s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.156905248s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:21:14.120326162 +0000 UTC m=+3122.810094468" watchObservedRunningTime="2026-10-31 08:21:14.121984009 +0000 UTC m=+3122.811752301"
Oct 31 08:21:15.106505 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:15.106469    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 31 08:21:15.903975 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:15.903939    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:21:19.400464 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:21:19.400416    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:21:20.111270 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:20.111238    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:20.111891 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:20.111867    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.83:8080: connect: connection refused"
Oct 31 08:21:25.904338 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:25.904293    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:21:30.112777 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:30.112739    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:35.904672 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:35.904629    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:21:38.059896 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.059859    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"]
Oct 31 08:21:38.060396 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.060157    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kserve-container" containerID="cri-o://ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f" gracePeriod=30
Oct 31 08:21:38.060396 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.060176    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kube-rbac-proxy" containerID="cri-o://7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d" gracePeriod=30
Oct 31 08:21:38.218790 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.218758    4913 generic.go:356] "Generic (PLEG): container finished" podID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerID="7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d" exitCode=0
Oct 31 08:21:38.218957 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.218834    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" event={"ID":"ad138610-d7a2-4d79-9f04-2b1cd7f20b46","Type":"ContainerDied","Data":"7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d"}
Oct 31 08:21:38.802366 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.802339    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:38.927388 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.927355    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-api-access-zvtc8\" (UniqueName: \"kubernetes.io/projected/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-api-access-zvtc8\") pod \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") "
Oct 31 08:21:38.927550 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.927427    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-proxy-tls\") pod \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") "
Oct 31 08:21:38.927550 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.927473    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-rbac-proxy-sar-config\") pod \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") "
Oct 31 08:21:38.927550 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.927541    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kserve-provision-location\") pod \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") "
Oct 31 08:21:38.927730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.927611    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-cabundle-cert\") pod \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\" (UID: \"ad138610-d7a2-4d79-9f04-2b1cd7f20b46\") "
Oct 31 08:21:38.927939 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.927899    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-rbac-proxy-sar-config" pod "ad138610-d7a2-4d79-9f04-2b1cd7f20b46" (UID: "ad138610-d7a2-4d79-9f04-2b1cd7f20b46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:21:38.927939 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.927929    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kserve-provision-location" pod "ad138610-d7a2-4d79-9f04-2b1cd7f20b46" (UID: "ad138610-d7a2-4d79-9f04-2b1cd7f20b46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:21:38.928154 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.928132    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-cabundle-cert" pod "ad138610-d7a2-4d79-9f04-2b1cd7f20b46" (UID: "ad138610-d7a2-4d79-9f04-2b1cd7f20b46"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:21:38.929445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.929422    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-api-access-zvtc8" pod "ad138610-d7a2-4d79-9f04-2b1cd7f20b46" (UID: "ad138610-d7a2-4d79-9f04-2b1cd7f20b46"). InnerVolumeSpecName "kube-api-access-zvtc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:21:38.929591 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:38.929571    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-proxy-tls" pod "ad138610-d7a2-4d79-9f04-2b1cd7f20b46" (UID: "ad138610-d7a2-4d79-9f04-2b1cd7f20b46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:21:39.028676 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.028634    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:39.028676 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.028674    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:39.028849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.028685    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvtc8\" (UniqueName: \"kubernetes.io/projected/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-api-access-zvtc8\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:39.028849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.028695    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:39.028849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.028705    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad138610-d7a2-4d79-9f04-2b1cd7f20b46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:21:39.225331 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.225238    4913 generic.go:356] "Generic (PLEG): container finished" podID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerID="ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f" exitCode=0
Oct 31 08:21:39.225331 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.225295    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"
Oct 31 08:21:39.225778 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.225376    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" event={"ID":"ad138610-d7a2-4d79-9f04-2b1cd7f20b46","Type":"ContainerDied","Data":"ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f"}
Oct 31 08:21:39.225778 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.225451    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8" event={"ID":"ad138610-d7a2-4d79-9f04-2b1cd7f20b46","Type":"ContainerDied","Data":"a11a1c983d1a1ff4a2797df93f58ec1ff9adc6f411f7f00e46c66b3be591c193"}
Oct 31 08:21:39.225778 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.225502    4913 scope.go:122] "RemoveContainer" containerID="7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d"
Oct 31 08:21:39.234559 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.234538    4913 scope.go:122] "RemoveContainer" containerID="ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f"
Oct 31 08:21:39.241881 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.241865    4913 scope.go:122] "RemoveContainer" containerID="1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa"
Oct 31 08:21:39.246359 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.246338    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"]
Oct 31 08:21:39.249629 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.249605    4913 scope.go:122] "RemoveContainer" containerID="7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d"
Oct 31 08:21:39.249894 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:21:39.249877    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d\": container with ID starting with 7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d not found: ID does not exist" containerID="7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d"
Oct 31 08:21:39.249942 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.249902    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d"} err="failed to get container status \"7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d\": rpc error: code = NotFound desc = could not find container \"7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d\": container with ID starting with 7bd68163c570d2a74be259d4253efbc63add938f77091a6f51d03aed66b4ec2d not found: ID does not exist"
Oct 31 08:21:39.249942 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.249919    4913 scope.go:122] "RemoveContainer" containerID="ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f"
Oct 31 08:21:39.250139 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:21:39.250124    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f\": container with ID starting with ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f not found: ID does not exist" containerID="ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f"
Oct 31 08:21:39.250219 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.250169    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f"} err="failed to get container status \"ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f\": rpc error: code = NotFound desc = could not find container \"ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f\": container with ID starting with ca4329bbffe66194e284c90dd1b7384aa8ffba51d99fbe7deff7bba076c0651f not found: ID does not exist"
Oct 31 08:21:39.250283 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.250223    4913 scope.go:122] "RemoveContainer" containerID="1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa"
Oct 31 08:21:39.250559 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:21:39.250534    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa\": container with ID starting with 1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa not found: ID does not exist" containerID="1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa"
Oct 31 08:21:39.250679 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.250568    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa"} err="failed to get container status \"1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa\": rpc error: code = NotFound desc = could not find container \"1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa\": container with ID starting with 1d3d6ecb6e756ab3adedf2dd53ebca98aa4a9f85522d48716d73ee8671b868aa not found: ID does not exist"
Oct 31 08:21:39.251966 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.251945    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-13001498cc/isvc-lightgbm-v2-kserve-predictor-5f4cc89959-gl6b8"]
Oct 31 08:21:39.419601 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:39.419570    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" path="/var/lib/kubelet/pods/ad138610-d7a2-4d79-9f04-2b1cd7f20b46/volumes"
Oct 31 08:21:45.903987 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:45.903948    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.82:8080: connect: connection refused"
Oct 31 08:21:49.399165 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:21:49.399127    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:21:55.905155 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:21:55.905123    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:22:05.879947 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:05.879909    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"]
Oct 31 08:22:05.880339 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:05.880208    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container" containerID="cri-o://959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3" gracePeriod=30
Oct 31 08:22:05.880609 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:05.880586    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kube-rbac-proxy" containerID="cri-o://253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743" gracePeriod=30
Oct 31 08:22:05.899233 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:05.899197    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.82:8643/healthz\": dial tcp 10.132.0.82:8643: connect: connection refused"
Oct 31 08:22:06.336163 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:06.336128    4913 generic.go:356] "Generic (PLEG): container finished" podID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerID="253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743" exitCode=0
Oct 31 08:22:06.336365 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:06.336218    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" event={"ID":"6914600e-79c0-4f12-8e30-f2f531feaa44","Type":"ContainerDied","Data":"253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743"}
Oct 31 08:22:07.072603 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.072561    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"]
Oct 31 08:22:07.073133 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.073070    4913 cpu_manager.go:414] "containerMap: removing container" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="storage-initializer"
Oct 31 08:22:07.073133 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.073089    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="storage-initializer"
Oct 31 08:22:07.073133 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.073102    4913 cpu_manager.go:414] "containerMap: removing container" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kube-rbac-proxy"
Oct 31 08:22:07.073133 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.073109    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kube-rbac-proxy"
Oct 31 08:22:07.073133 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.073128    4913 cpu_manager.go:414] "containerMap: removing container" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kserve-container"
Oct 31 08:22:07.073133 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.073137    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kserve-container"
Oct 31 08:22:07.073563 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.073260    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kube-rbac-proxy"
Oct 31 08:22:07.073563 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.073277    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad138610-d7a2-4d79-9f04-2b1cd7f20b46" containerName="kserve-container"
Oct 31 08:22:07.076649 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.076627    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.085442 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.085416    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Oct 31 08:22:07.085697 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.085680    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Oct 31 08:22:07.087129 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.087107    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"]
Oct 31 08:22:07.160352 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.160320    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abf0677-6f81-4929-8c1b-10f46320b739-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.160523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.160365    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4ct\" (UniqueName: \"kubernetes.io/projected/4abf0677-6f81-4929-8c1b-10f46320b739-kube-api-access-gv4ct\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.160523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.160487    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abf0677-6f81-4929-8c1b-10f46320b739-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.160523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.160510    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abf0677-6f81-4929-8c1b-10f46320b739-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.260858 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.260812    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abf0677-6f81-4929-8c1b-10f46320b739-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.260858 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.260859    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abf0677-6f81-4929-8c1b-10f46320b739-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.261093 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.260895    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abf0677-6f81-4929-8c1b-10f46320b739-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.261093 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.260918    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4ct\" (UniqueName: \"kubernetes.io/projected/4abf0677-6f81-4929-8c1b-10f46320b739-kube-api-access-gv4ct\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.261404 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.261383    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abf0677-6f81-4929-8c1b-10f46320b739-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.261659 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.261639    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abf0677-6f81-4929-8c1b-10f46320b739-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.263330 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.263315    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abf0677-6f81-4929-8c1b-10f46320b739-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.270944 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.270919    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4ct\" (UniqueName: \"kubernetes.io/projected/4abf0677-6f81-4929-8c1b-10f46320b739-kube-api-access-gv4ct\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") " pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.394331 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.394248    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:07.672523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:07.672487    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"]
Oct 31 08:22:07.680539 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:22:07.680508    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abf0677_6f81_4929_8c1b_10f46320b739.slice/crio-8ce52c0a8cb8ca9bbbb4cd8b6cc2bcda1ee88b64012ac982d76820a4ba0d3fa0 WatchSource:0}: Error finding container 8ce52c0a8cb8ca9bbbb4cd8b6cc2bcda1ee88b64012ac982d76820a4ba0d3fa0: Status 404 returned error can't find the container with id 8ce52c0a8cb8ca9bbbb4cd8b6cc2bcda1ee88b64012ac982d76820a4ba0d3fa0
Oct 31 08:22:08.346333 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:08.346286    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5" event={"ID":"4abf0677-6f81-4929-8c1b-10f46320b739","Type":"ContainerStarted","Data":"14df1506799da5b7e30d9b382537cd7181d080812cc19abc3074248df905e51f"}
Oct 31 08:22:08.346748 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:08.346374    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5" event={"ID":"4abf0677-6f81-4929-8c1b-10f46320b739","Type":"ContainerStarted","Data":"8ce52c0a8cb8ca9bbbb4cd8b6cc2bcda1ee88b64012ac982d76820a4ba0d3fa0"}
Oct 31 08:22:10.927264 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.927240    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:22:10.991689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.991590    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-api-access-lsxf8\" (UniqueName: \"kubernetes.io/projected/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-api-access-lsxf8\") pod \"6914600e-79c0-4f12-8e30-f2f531feaa44\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") "
Oct 31 08:22:10.991689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.991651    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6914600e-79c0-4f12-8e30-f2f531feaa44-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6914600e-79c0-4f12-8e30-f2f531feaa44-proxy-tls\") pod \"6914600e-79c0-4f12-8e30-f2f531feaa44\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") "
Oct 31 08:22:10.991689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.991676    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6914600e-79c0-4f12-8e30-f2f531feaa44-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6914600e-79c0-4f12-8e30-f2f531feaa44-kserve-provision-location\") pod \"6914600e-79c0-4f12-8e30-f2f531feaa44\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") "
Oct 31 08:22:10.991978 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.991718    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-cabundle-cert\") pod \"6914600e-79c0-4f12-8e30-f2f531feaa44\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") "
Oct 31 08:22:10.991978 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.991748    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-rbac-proxy-sar-config\") pod \"6914600e-79c0-4f12-8e30-f2f531feaa44\" (UID: \"6914600e-79c0-4f12-8e30-f2f531feaa44\") "
Oct 31 08:22:10.991978 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.991956    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6914600e-79c0-4f12-8e30-f2f531feaa44-kserve-provision-location" pod "6914600e-79c0-4f12-8e30-f2f531feaa44" (UID: "6914600e-79c0-4f12-8e30-f2f531feaa44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:22:10.992145 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.992095    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-cabundle-cert" pod "6914600e-79c0-4f12-8e30-f2f531feaa44" (UID: "6914600e-79c0-4f12-8e30-f2f531feaa44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:22:10.992145 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.992132    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-rbac-proxy-sar-config" pod "6914600e-79c0-4f12-8e30-f2f531feaa44" (UID: "6914600e-79c0-4f12-8e30-f2f531feaa44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:22:10.993754 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.993731    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-api-access-lsxf8" pod "6914600e-79c0-4f12-8e30-f2f531feaa44" (UID: "6914600e-79c0-4f12-8e30-f2f531feaa44"). InnerVolumeSpecName "kube-api-access-lsxf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:22:10.993841 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:10.993772    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6914600e-79c0-4f12-8e30-f2f531feaa44-proxy-tls" pod "6914600e-79c0-4f12-8e30-f2f531feaa44" (UID: "6914600e-79c0-4f12-8e30-f2f531feaa44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:22:11.092866 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.092826    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:11.092866 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.092857    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:11.092866 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.092867    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsxf8\" (UniqueName: \"kubernetes.io/projected/6914600e-79c0-4f12-8e30-f2f531feaa44-kube-api-access-lsxf8\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:11.092866 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.092878    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6914600e-79c0-4f12-8e30-f2f531feaa44-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:11.093133 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.092887    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6914600e-79c0-4f12-8e30-f2f531feaa44-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:11.361432 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.361387    4913 generic.go:356] "Generic (PLEG): container finished" podID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerID="959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3" exitCode=0
Oct 31 08:22:11.361616 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.361454    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"
Oct 31 08:22:11.361616 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.361471    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" event={"ID":"6914600e-79c0-4f12-8e30-f2f531feaa44","Type":"ContainerDied","Data":"959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3"}
Oct 31 08:22:11.361616 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.361561    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" event={"ID":"6914600e-79c0-4f12-8e30-f2f531feaa44","Type":"ContainerDied","Data":"f050384b64dd98160fc437783dda37d32656246c862be52878ec113e2bf3a440"}
Oct 31 08:22:11.361616 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.361583    4913 scope.go:122] "RemoveContainer" containerID="253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743"
Oct 31 08:22:11.363228 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.363205    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5_4abf0677-6f81-4929-8c1b-10f46320b739/storage-initializer/0.log"
Oct 31 08:22:11.363330 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.363246    4913 generic.go:356] "Generic (PLEG): container finished" podID="4abf0677-6f81-4929-8c1b-10f46320b739" containerID="14df1506799da5b7e30d9b382537cd7181d080812cc19abc3074248df905e51f" exitCode=1
Oct 31 08:22:11.363330 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.363278    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5" event={"ID":"4abf0677-6f81-4929-8c1b-10f46320b739","Type":"ContainerDied","Data":"14df1506799da5b7e30d9b382537cd7181d080812cc19abc3074248df905e51f"}
Oct 31 08:22:11.398506 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.398483    4913 scope.go:122] "RemoveContainer" containerID="959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3"
Oct 31 08:22:11.430819 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.430791    4913 scope.go:122] "RemoveContainer" containerID="731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1"
Oct 31 08:22:11.438844 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.438821    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"]
Oct 31 08:22:11.442844 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.442820    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns"]
Oct 31 08:22:11.449241 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.449219    4913 scope.go:122] "RemoveContainer" containerID="253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743"
Oct 31 08:22:11.449619 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:22:11.449588    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743\": container with ID starting with 253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743 not found: ID does not exist" containerID="253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743"
Oct 31 08:22:11.449696 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.449631    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743"} err="failed to get container status \"253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743\": rpc error: code = NotFound desc = could not find container \"253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743\": container with ID starting with 253c2b65342315ee836a7d3e81e4125269309d56c0becd030711be7b77ade743 not found: ID does not exist"
Oct 31 08:22:11.449696 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.449656    4913 scope.go:122] "RemoveContainer" containerID="959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3"
Oct 31 08:22:11.449981 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:22:11.449958    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3\": container with ID starting with 959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3 not found: ID does not exist" containerID="959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3"
Oct 31 08:22:11.450024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.449990    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3"} err="failed to get container status \"959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3\": rpc error: code = NotFound desc = could not find container \"959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3\": container with ID starting with 959d249e045123e0643ca5e2ef8fbbc52e9a3b6ab5d24ac300900baceb0f99c3 not found: ID does not exist"
Oct 31 08:22:11.450024 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.450005    4913 scope.go:122] "RemoveContainer" containerID="731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1"
Oct 31 08:22:11.450362 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:22:11.450335    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1\": container with ID starting with 731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1 not found: ID does not exist" containerID="731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1"
Oct 31 08:22:11.450466 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.450368    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1"} err="failed to get container status \"731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1\": rpc error: code = NotFound desc = could not find container \"731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1\": container with ID starting with 731f5dfd6a8c6d4c7855948e002536ace6a94e444652019f75dc5d1625ad55b1 not found: ID does not exist"
Oct 31 08:22:11.899576 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:11.899533    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-pass-predictor-6cfd86d494-xh2ns" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.82:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:22:12.369062 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:12.369034    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5_4abf0677-6f81-4929-8c1b-10f46320b739/storage-initializer/0.log"
Oct 31 08:22:12.369466 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:12.369081    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5" event={"ID":"4abf0677-6f81-4929-8c1b-10f46320b739","Type":"ContainerStarted","Data":"8af0c9ffdb130becaf46aaf193a2cef95d9acf816f2609da077725db67605518"}
Oct 31 08:22:13.420041 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:13.420004    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" path="/var/lib/kubelet/pods/6914600e-79c0-4f12-8e30-f2f531feaa44/volumes"
Oct 31 08:22:17.071911 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.071880    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"]
Oct 31 08:22:17.072330 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.072105    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5" podUID="4abf0677-6f81-4929-8c1b-10f46320b739" containerName="storage-initializer" containerID="cri-o://8af0c9ffdb130becaf46aaf193a2cef95d9acf816f2609da077725db67605518" gracePeriod=30
Oct 31 08:22:17.390474 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.390448    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5_4abf0677-6f81-4929-8c1b-10f46320b739/storage-initializer/1.log"
Oct 31 08:22:17.390874 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.390856    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5_4abf0677-6f81-4929-8c1b-10f46320b739/storage-initializer/0.log"
Oct 31 08:22:17.391007 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.390894    4913 generic.go:356] "Generic (PLEG): container finished" podID="4abf0677-6f81-4929-8c1b-10f46320b739" containerID="8af0c9ffdb130becaf46aaf193a2cef95d9acf816f2609da077725db67605518" exitCode=1
Oct 31 08:22:17.391007 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.390935    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5" event={"ID":"4abf0677-6f81-4929-8c1b-10f46320b739","Type":"ContainerDied","Data":"8af0c9ffdb130becaf46aaf193a2cef95d9acf816f2609da077725db67605518"}
Oct 31 08:22:17.391118 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.391005    4913 scope.go:122] "RemoveContainer" containerID="14df1506799da5b7e30d9b382537cd7181d080812cc19abc3074248df905e51f"
Oct 31 08:22:17.411131 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.411108    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5_4abf0677-6f81-4929-8c1b-10f46320b739/storage-initializer/1.log"
Oct 31 08:22:17.414279 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.414258    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:17.442682 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.442653    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf0677-6f81-4929-8c1b-10f46320b739-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abf0677-6f81-4929-8c1b-10f46320b739-kserve-provision-location\") pod \"4abf0677-6f81-4929-8c1b-10f46320b739\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") "
Oct 31 08:22:17.442849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.442687    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abf0677-6f81-4929-8c1b-10f46320b739-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abf0677-6f81-4929-8c1b-10f46320b739-kube-rbac-proxy-sar-config\") pod \"4abf0677-6f81-4929-8c1b-10f46320b739\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") "
Oct 31 08:22:17.442849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.442715    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abf0677-6f81-4929-8c1b-10f46320b739-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abf0677-6f81-4929-8c1b-10f46320b739-proxy-tls\") pod \"4abf0677-6f81-4929-8c1b-10f46320b739\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") "
Oct 31 08:22:17.442849 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.442769    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abf0677-6f81-4929-8c1b-10f46320b739-kube-api-access-gv4ct\" (UniqueName: \"kubernetes.io/projected/4abf0677-6f81-4929-8c1b-10f46320b739-kube-api-access-gv4ct\") pod \"4abf0677-6f81-4929-8c1b-10f46320b739\" (UID: \"4abf0677-6f81-4929-8c1b-10f46320b739\") "
Oct 31 08:22:17.443083 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.442907    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf0677-6f81-4929-8c1b-10f46320b739-kserve-provision-location" pod "4abf0677-6f81-4929-8c1b-10f46320b739" (UID: "4abf0677-6f81-4929-8c1b-10f46320b739"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:22:17.443083 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.443067    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abf0677-6f81-4929-8c1b-10f46320b739-kube-rbac-proxy-sar-config" pod "4abf0677-6f81-4929-8c1b-10f46320b739" (UID: "4abf0677-6f81-4929-8c1b-10f46320b739"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:22:17.444833 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.444812    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abf0677-6f81-4929-8c1b-10f46320b739-proxy-tls" pod "4abf0677-6f81-4929-8c1b-10f46320b739" (UID: "4abf0677-6f81-4929-8c1b-10f46320b739"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:22:17.444937 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.444842    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abf0677-6f81-4929-8c1b-10f46320b739-kube-api-access-gv4ct" pod "4abf0677-6f81-4929-8c1b-10f46320b739" (UID: "4abf0677-6f81-4929-8c1b-10f46320b739"). InnerVolumeSpecName "kube-api-access-gv4ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:22:17.543426 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.543378    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv4ct\" (UniqueName: \"kubernetes.io/projected/4abf0677-6f81-4929-8c1b-10f46320b739-kube-api-access-gv4ct\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:17.543426 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.543421    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abf0677-6f81-4929-8c1b-10f46320b739-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:17.543643 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.543432    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abf0677-6f81-4929-8c1b-10f46320b739-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:17.543643 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:17.543457    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abf0677-6f81-4929-8c1b-10f46320b739-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:22:18.397437 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:18.397409    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-9226a10e02_isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5_4abf0677-6f81-4929-8c1b-10f46320b739/storage-initializer/1.log"
Oct 31 08:22:18.397901 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:18.397471    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5" event={"ID":"4abf0677-6f81-4929-8c1b-10f46320b739","Type":"ContainerDied","Data":"8ce52c0a8cb8ca9bbbb4cd8b6cc2bcda1ee88b64012ac982d76820a4ba0d3fa0"}
Oct 31 08:22:18.397901 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:18.397505    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"
Oct 31 08:22:18.397901 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:18.397529    4913 scope.go:122] "RemoveContainer" containerID="8af0c9ffdb130becaf46aaf193a2cef95d9acf816f2609da077725db67605518"
Oct 31 08:22:18.435023 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:18.434980    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"]
Oct 31 08:22:18.442884 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:18.437236    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-sklearn-s3-tls-serving-fail-predictor-6694bf45b8-hwmj5"]
Oct 31 08:22:19.400458 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:22:19.400420    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:22:19.420353 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:19.420320    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abf0677-6f81-4929-8c1b-10f46320b739" path="/var/lib/kubelet/pods/4abf0677-6f81-4929-8c1b-10f46320b739/volumes"
Oct 31 08:22:20.514305 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514270    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"]
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514574    4913 cpu_manager.go:414] "containerMap: removing container" podUID="4abf0677-6f81-4929-8c1b-10f46320b739" containerName="storage-initializer"
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514585    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abf0677-6f81-4929-8c1b-10f46320b739" containerName="storage-initializer"
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514612    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container"
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514618    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container"
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514627    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="storage-initializer"
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514632    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="storage-initializer"
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514641    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kube-rbac-proxy"
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514645    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kube-rbac-proxy"
Oct 31 08:22:20.514693 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514698    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kserve-container"
Oct 31 08:22:20.515010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514707    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6914600e-79c0-4f12-8e30-f2f531feaa44" containerName="kube-rbac-proxy"
Oct 31 08:22:20.515010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514714    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abf0677-6f81-4929-8c1b-10f46320b739" containerName="storage-initializer"
Oct 31 08:22:20.515010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514722    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abf0677-6f81-4929-8c1b-10f46320b739" containerName="storage-initializer"
Oct 31 08:22:20.515010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514771    4913 cpu_manager.go:414] "containerMap: removing container" podUID="4abf0677-6f81-4929-8c1b-10f46320b739" containerName="storage-initializer"
Oct 31 08:22:20.515010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.514777    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abf0677-6f81-4929-8c1b-10f46320b739" containerName="storage-initializer"
Oct 31 08:22:20.519423 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.519400    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.522721 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.522698    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Oct 31 08:22:20.522847 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.522760    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:22:20.525761 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.525741    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:22:20.527732 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.527707    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"]
Oct 31 08:22:20.529451 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.529422    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:22:20.529618 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.529599    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:22:20.529755 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.529738    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Oct 31 08:22:20.569638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.569599    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.569800 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.569654    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.569840 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.569820    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.569887 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.569874    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-proxy-tls\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.569969 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.569949    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxcxm\" (UniqueName: \"kubernetes.io/projected/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-api-access-nxcxm\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.623082 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.623048    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:22:20.671408 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.671370    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.671590 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.671422    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.671590 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.671465    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.671590 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.671495    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-proxy-tls\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.671590 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.671539    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxcxm\" (UniqueName: \"kubernetes.io/projected/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-api-access-nxcxm\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.671894 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.671866    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.672139 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.672120    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.672247 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.672228    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.674014 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.673996    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-proxy-tls\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.679004 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.678974    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxcxm\" (UniqueName: \"kubernetes.io/projected/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-api-access-nxcxm\") pod \"isvc-paddle-runtime-predictor-645c546cf7-rh24r\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") " pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:20.838395 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:20.838309    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:21.113021 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:21.112993    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"]
Oct 31 08:22:21.115828 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:22:21.115801    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb6c2d1_8fe3_480e_90c5_14695e48fb79.slice/crio-2417a55a351793ac4c07616354f5ac3e9b47b3f59280fe3d2bac956e686d30ef WatchSource:0}: Error finding container 2417a55a351793ac4c07616354f5ac3e9b47b3f59280fe3d2bac956e686d30ef: Status 404 returned error can't find the container with id 2417a55a351793ac4c07616354f5ac3e9b47b3f59280fe3d2bac956e686d30ef
Oct 31 08:22:21.420357 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:21.420319    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" event={"ID":"2cb6c2d1-8fe3-480e-90c5-14695e48fb79","Type":"ContainerStarted","Data":"71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746"}
Oct 31 08:22:21.420513 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:21.420401    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" event={"ID":"2cb6c2d1-8fe3-480e-90c5-14695e48fb79","Type":"ContainerStarted","Data":"2417a55a351793ac4c07616354f5ac3e9b47b3f59280fe3d2bac956e686d30ef"}
Oct 31 08:22:27.438736 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:27.438697    4913 generic.go:356] "Generic (PLEG): container finished" podID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerID="71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746" exitCode=0
Oct 31 08:22:27.439314 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:27.438771    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" event={"ID":"2cb6c2d1-8fe3-480e-90c5-14695e48fb79","Type":"ContainerDied","Data":"71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746"}
Oct 31 08:22:27.440115 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:27.440100    4913 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 08:22:39.497254 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:39.497221    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" event={"ID":"2cb6c2d1-8fe3-480e-90c5-14695e48fb79","Type":"ContainerStarted","Data":"5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3"}
Oct 31 08:22:39.497720 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:39.497282    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" event={"ID":"2cb6c2d1-8fe3-480e-90c5-14695e48fb79","Type":"ContainerStarted","Data":"89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9"}
Oct 31 08:22:39.497720 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:39.497488    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:39.516701 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:39.516652    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podStartSLOduration=2.624104143 podStartE2EDuration="19.516637588s" totalImagesPullingTime="11.36858492s" totalInitContainerRuntime="5.523948525s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:22:39.515131738 +0000 UTC m=+3208.204900034" watchObservedRunningTime="2026-10-31 08:22:39.516637588 +0000 UTC m=+3208.206405880"
Oct 31 08:22:40.501299 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:40.501268    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:40.502639 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:40.502613    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 31 08:22:41.504950 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:41.504909    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 31 08:22:46.509778 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:46.509744    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:22:46.510452 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:46.510419    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 31 08:22:49.400585 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:22:49.400543    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:22:56.510740 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:22:56.510700    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 31 08:23:06.510414 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:06.510370    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 31 08:23:16.510273 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:16.510232    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.85:8080: connect: connection refused"
Oct 31 08:23:19.399223 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:23:19.399164    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:23:26.511349 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:26.511319    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:23:32.285901 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:32.285863    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"]
Oct 31 08:23:32.286335 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:32.286095    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container" containerID="cri-o://89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9" gracePeriod=30
Oct 31 08:23:32.286434 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:32.286412    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kube-rbac-proxy" containerID="cri-o://5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3" gracePeriod=30
Oct 31 08:23:32.722331 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:32.722299    4913 generic.go:356] "Generic (PLEG): container finished" podID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerID="5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3" exitCode=0
Oct 31 08:23:32.722499 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:32.722336    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" event={"ID":"2cb6c2d1-8fe3-480e-90c5-14695e48fb79","Type":"ContainerDied","Data":"5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3"}
Oct 31 08:23:35.327114 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.327086    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:23:35.386346 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.386256    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-api-access-nxcxm\" (UniqueName: \"kubernetes.io/projected/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-api-access-nxcxm\") pod \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") "
Oct 31 08:23:35.386525 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.386344    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-cabundle-cert\") pod \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") "
Oct 31 08:23:35.386525 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.386419    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-proxy-tls\") pod \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") "
Oct 31 08:23:35.386525 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.386458    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kserve-provision-location\") pod \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") "
Oct 31 08:23:35.386525 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.386511    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-rbac-proxy-sar-config\") pod \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\" (UID: \"2cb6c2d1-8fe3-480e-90c5-14695e48fb79\") "
Oct 31 08:23:35.386828 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.386701    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-cabundle-cert" pod "2cb6c2d1-8fe3-480e-90c5-14695e48fb79" (UID: "2cb6c2d1-8fe3-480e-90c5-14695e48fb79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:23:35.386958 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.386906    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-rbac-proxy-sar-config" pod "2cb6c2d1-8fe3-480e-90c5-14695e48fb79" (UID: "2cb6c2d1-8fe3-480e-90c5-14695e48fb79"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:23:35.388574 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.388540    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-proxy-tls" pod "2cb6c2d1-8fe3-480e-90c5-14695e48fb79" (UID: "2cb6c2d1-8fe3-480e-90c5-14695e48fb79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:23:35.388684 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.388605    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-api-access-nxcxm" pod "2cb6c2d1-8fe3-480e-90c5-14695e48fb79" (UID: "2cb6c2d1-8fe3-480e-90c5-14695e48fb79"). InnerVolumeSpecName "kube-api-access-nxcxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:23:35.395984 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.395952    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kserve-provision-location" pod "2cb6c2d1-8fe3-480e-90c5-14695e48fb79" (UID: "2cb6c2d1-8fe3-480e-90c5-14695e48fb79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:23:35.487398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.487355    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:23:35.487398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.487390    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxcxm\" (UniqueName: \"kubernetes.io/projected/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kube-api-access-nxcxm\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:23:35.487398 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.487407    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:23:35.487658 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.487431    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:23:35.487658 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.487441    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb6c2d1-8fe3-480e-90c5-14695e48fb79-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:23:35.736515 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.736476    4913 generic.go:356] "Generic (PLEG): container finished" podID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerID="89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9" exitCode=0
Oct 31 08:23:35.736689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.736539    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"
Oct 31 08:23:35.736689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.736581    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" event={"ID":"2cb6c2d1-8fe3-480e-90c5-14695e48fb79","Type":"ContainerDied","Data":"89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9"}
Oct 31 08:23:35.736689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.736633    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r" event={"ID":"2cb6c2d1-8fe3-480e-90c5-14695e48fb79","Type":"ContainerDied","Data":"2417a55a351793ac4c07616354f5ac3e9b47b3f59280fe3d2bac956e686d30ef"}
Oct 31 08:23:35.736689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.736660    4913 scope.go:122] "RemoveContainer" containerID="5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3"
Oct 31 08:23:35.745103 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.745085    4913 scope.go:122] "RemoveContainer" containerID="89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9"
Oct 31 08:23:35.753078 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.753058    4913 scope.go:122] "RemoveContainer" containerID="71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746"
Oct 31 08:23:35.753421 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.753393    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"]
Oct 31 08:23:35.757093 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.757069    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-runtime-predictor-645c546cf7-rh24r"]
Oct 31 08:23:35.760913 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.760896    4913 scope.go:122] "RemoveContainer" containerID="5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3"
Oct 31 08:23:35.761161 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:23:35.761141    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3\": container with ID starting with 5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3 not found: ID does not exist" containerID="5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3"
Oct 31 08:23:35.761253 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.761173    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3"} err="failed to get container status \"5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3\": rpc error: code = NotFound desc = could not find container \"5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3\": container with ID starting with 5191d9d0a185ac41632393a563db18f4b8f32979d88e433879412b94c5b80ea3 not found: ID does not exist"
Oct 31 08:23:35.761253 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.761215    4913 scope.go:122] "RemoveContainer" containerID="89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9"
Oct 31 08:23:35.761472 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:23:35.761454    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9\": container with ID starting with 89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9 not found: ID does not exist" containerID="89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9"
Oct 31 08:23:35.761513 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.761483    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9"} err="failed to get container status \"89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9\": rpc error: code = NotFound desc = could not find container \"89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9\": container with ID starting with 89cfab3ce2f0a93b441210d24d14f18f701913685b4a54006687ae84399466a9 not found: ID does not exist"
Oct 31 08:23:35.761513 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.761501    4913 scope.go:122] "RemoveContainer" containerID="71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746"
Oct 31 08:23:35.761756 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:23:35.761737    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746\": container with ID starting with 71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746 not found: ID does not exist" containerID="71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746"
Oct 31 08:23:35.761810 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:35.761760    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746"} err="failed to get container status \"71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746\": rpc error: code = NotFound desc = could not find container \"71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746\": container with ID starting with 71e637489460433ca57a2acdd0fdeefd4eb1a1d8b503950dc7f9b2a869bd6746 not found: ID does not exist"
Oct 31 08:23:37.420131 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.420098    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" path="/var/lib/kubelet/pods/2cb6c2d1-8fe3-480e-90c5-14695e48fb79/volumes"
Oct 31 08:23:37.685889 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.685853    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"]
Oct 31 08:23:37.686217 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.686204    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="storage-initializer"
Oct 31 08:23:37.686275 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.686218    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="storage-initializer"
Oct 31 08:23:37.686275 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.686235    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kube-rbac-proxy"
Oct 31 08:23:37.686275 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.686242    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kube-rbac-proxy"
Oct 31 08:23:37.686275 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.686259    4913 cpu_manager.go:414] "containerMap: removing container" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container"
Oct 31 08:23:37.686275 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.686267    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container"
Oct 31 08:23:37.686429 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.686323    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kube-rbac-proxy"
Oct 31 08:23:37.686429 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.686335    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb6c2d1-8fe3-480e-90c5-14695e48fb79" containerName="kserve-container"
Oct 31 08:23:37.690823 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.690803    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.698308 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.698278    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"]
Oct 31 08:23:37.699791 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.699769    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:23:37.700614 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.700498    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Oct 31 08:23:37.700614 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.700559    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:23:37.700614 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.700570    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:23:37.700614 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.700570    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 31 08:23:37.700614 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.700573    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:23:37.700924 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.700623    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:23:37.806689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.806647    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.806689 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.806688    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh67m\" (UniqueName: \"kubernetes.io/projected/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-api-access-qh67m\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.806917 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.806715    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.806917 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.806797    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b7521bc-ac6a-4159-b363-b2715425b93b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.806917 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.806866    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b7521bc-ac6a-4159-b363-b2715425b93b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.908088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.908050    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.908088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.908086    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh67m\" (UniqueName: \"kubernetes.io/projected/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-api-access-qh67m\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.908360 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.908106    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.908360 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.908148    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b7521bc-ac6a-4159-b363-b2715425b93b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.908360 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.908177    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b7521bc-ac6a-4159-b363-b2715425b93b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.908648 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.908623    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b7521bc-ac6a-4159-b363-b2715425b93b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.908877 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.908857    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.908915 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.908855    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.910606 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.910590    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b7521bc-ac6a-4159-b363-b2715425b93b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:37.915999 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:37.915974    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh67m\" (UniqueName: \"kubernetes.io/projected/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-api-access-qh67m\") pod \"isvc-paddle-v2-kserve-predictor-686c49d88f-784sj\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") " pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:38.009130 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:38.009042    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:38.282521 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:38.282446    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"]
Oct 31 08:23:38.285709 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:23:38.285685    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7521bc_ac6a_4159_b363_b2715425b93b.slice/crio-edb4ca092aea45cdfc8b648bfdfed69e68fdceecb070b881270f0579a7806001 WatchSource:0}: Error finding container edb4ca092aea45cdfc8b648bfdfed69e68fdceecb070b881270f0579a7806001: Status 404 returned error can't find the container with id edb4ca092aea45cdfc8b648bfdfed69e68fdceecb070b881270f0579a7806001
Oct 31 08:23:38.752544 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:38.752508    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" event={"ID":"6b7521bc-ac6a-4159-b363-b2715425b93b","Type":"ContainerStarted","Data":"1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89"}
Oct 31 08:23:38.752939 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:38.752569    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" event={"ID":"6b7521bc-ac6a-4159-b363-b2715425b93b","Type":"ContainerStarted","Data":"edb4ca092aea45cdfc8b648bfdfed69e68fdceecb070b881270f0579a7806001"}
Oct 31 08:23:43.781052 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:43.781019    4913 generic.go:356] "Generic (PLEG): container finished" podID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerID="1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89" exitCode=0
Oct 31 08:23:43.781474 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:43.781060    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" event={"ID":"6b7521bc-ac6a-4159-b363-b2715425b93b","Type":"ContainerDied","Data":"1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89"}
Oct 31 08:23:44.788208 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:44.788155    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" event={"ID":"6b7521bc-ac6a-4159-b363-b2715425b93b","Type":"ContainerStarted","Data":"23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307"}
Oct 31 08:23:44.788619 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:44.788233    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" event={"ID":"6b7521bc-ac6a-4159-b363-b2715425b93b","Type":"ContainerStarted","Data":"01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683"}
Oct 31 08:23:44.788619 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:44.788422    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:44.809245 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:44.809170    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podStartSLOduration=3.357781025 podStartE2EDuration="7.809156878s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.451375853s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:23:44.807267171 +0000 UTC m=+3273.497035462" watchObservedRunningTime="2026-10-31 08:23:44.809156878 +0000 UTC m=+3273.498925172"
Oct 31 08:23:45.792822 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:45.792783    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:45.794196 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:45.794163    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 31 08:23:46.796930 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:46.796888    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 31 08:23:49.400619 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:23:49.400580    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:23:51.801692 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:51.801663    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:23:51.802342 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:23:51.802316    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 31 08:24:01.802898 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:01.802811    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 31 08:24:11.711219 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:11.711168    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:24:11.717788 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:11.717760    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:24:11.803151 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:11.803115    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 31 08:24:19.399196 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:24:19.399147    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:24:21.802771 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:21.802723    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 31 08:24:31.803383 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:31.803349    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:24:39.579734 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:39.579698    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"]
Oct 31 08:24:39.580373 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:39.579935    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container" containerID="cri-o://01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683" gracePeriod=30
Oct 31 08:24:39.580373 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:39.579976    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kube-rbac-proxy" containerID="cri-o://23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307" gracePeriod=30
Oct 31 08:24:40.022817 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:40.022783    4913 generic.go:356] "Generic (PLEG): container finished" podID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerID="23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307" exitCode=0
Oct 31 08:24:40.022986 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:40.022861    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" event={"ID":"6b7521bc-ac6a-4159-b363-b2715425b93b","Type":"ContainerDied","Data":"23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307"}
Oct 31 08:24:41.797901 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:41.797858    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.86:8643/healthz\": dial tcp 10.132.0.86:8643: connect: connection refused"
Oct 31 08:24:41.802790 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:41.802765    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.86:8080: connect: connection refused"
Oct 31 08:24:42.625342 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.625313    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:24:42.734081 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.733985    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-rbac-proxy-sar-config\") pod \"6b7521bc-ac6a-4159-b363-b2715425b93b\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") "
Oct 31 08:24:42.734389 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.734359    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7521bc-ac6a-4159-b363-b2715425b93b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b7521bc-ac6a-4159-b363-b2715425b93b-proxy-tls\") pod \"6b7521bc-ac6a-4159-b363-b2715425b93b\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") "
Oct 31 08:24:42.734524 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.734457    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-cabundle-cert\") pod \"6b7521bc-ac6a-4159-b363-b2715425b93b\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") "
Oct 31 08:24:42.734524 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.734488    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b7521bc-ac6a-4159-b363-b2715425b93b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b7521bc-ac6a-4159-b363-b2715425b93b-kserve-provision-location\") pod \"6b7521bc-ac6a-4159-b363-b2715425b93b\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") "
Oct 31 08:24:42.734671 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.734544    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-api-access-qh67m\" (UniqueName: \"kubernetes.io/projected/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-api-access-qh67m\") pod \"6b7521bc-ac6a-4159-b363-b2715425b93b\" (UID: \"6b7521bc-ac6a-4159-b363-b2715425b93b\") "
Oct 31 08:24:42.735229 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.735178    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-rbac-proxy-sar-config" pod "6b7521bc-ac6a-4159-b363-b2715425b93b" (UID: "6b7521bc-ac6a-4159-b363-b2715425b93b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:24:42.735383 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.735249    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-cabundle-cert" pod "6b7521bc-ac6a-4159-b363-b2715425b93b" (UID: "6b7521bc-ac6a-4159-b363-b2715425b93b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:24:42.737462 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.737434    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7521bc-ac6a-4159-b363-b2715425b93b-proxy-tls" pod "6b7521bc-ac6a-4159-b363-b2715425b93b" (UID: "6b7521bc-ac6a-4159-b363-b2715425b93b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:24:42.737933 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.737892    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-api-access-qh67m" pod "6b7521bc-ac6a-4159-b363-b2715425b93b" (UID: "6b7521bc-ac6a-4159-b363-b2715425b93b"). InnerVolumeSpecName "kube-api-access-qh67m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:24:42.746483 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.746447    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7521bc-ac6a-4159-b363-b2715425b93b-kserve-provision-location" pod "6b7521bc-ac6a-4159-b363-b2715425b93b" (UID: "6b7521bc-ac6a-4159-b363-b2715425b93b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:24:42.835106 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.835065    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh67m\" (UniqueName: \"kubernetes.io/projected/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-api-access-qh67m\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:42.835106 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.835099    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:42.835106 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.835109    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b7521bc-ac6a-4159-b363-b2715425b93b-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:42.835106 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.835120    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b7521bc-ac6a-4159-b363-b2715425b93b-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:42.835638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:42.835128    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b7521bc-ac6a-4159-b363-b2715425b93b-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:24:43.036321 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.036226    4913 generic.go:356] "Generic (PLEG): container finished" podID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerID="01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683" exitCode=0
Oct 31 08:24:43.036321 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.036298    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"
Oct 31 08:24:43.036321 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.036314    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" event={"ID":"6b7521bc-ac6a-4159-b363-b2715425b93b","Type":"ContainerDied","Data":"01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683"}
Oct 31 08:24:43.036604 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.036369    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj" event={"ID":"6b7521bc-ac6a-4159-b363-b2715425b93b","Type":"ContainerDied","Data":"edb4ca092aea45cdfc8b648bfdfed69e68fdceecb070b881270f0579a7806001"}
Oct 31 08:24:43.036604 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.036395    4913 scope.go:122] "RemoveContainer" containerID="23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307"
Oct 31 08:24:43.044887 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.044865    4913 scope.go:122] "RemoveContainer" containerID="01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683"
Oct 31 08:24:43.052363 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.052340    4913 scope.go:122] "RemoveContainer" containerID="1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89"
Oct 31 08:24:43.057638 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.057618    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"]
Oct 31 08:24:43.059685 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.059668    4913 scope.go:122] "RemoveContainer" containerID="23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307"
Oct 31 08:24:43.059999 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:24:43.059953    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307\": container with ID starting with 23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307 not found: ID does not exist" containerID="23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307"
Oct 31 08:24:43.059999 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.059990    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307"} err="failed to get container status \"23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307\": rpc error: code = NotFound desc = could not find container \"23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307\": container with ID starting with 23b68c2c83286325b057d7eac6079ccdd426e91d0d7d8ead587ffadb4de51307 not found: ID does not exist"
Oct 31 08:24:43.060533 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.060014    4913 scope.go:122] "RemoveContainer" containerID="01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683"
Oct 31 08:24:43.060533 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:24:43.060340    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683\": container with ID starting with 01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683 not found: ID does not exist" containerID="01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683"
Oct 31 08:24:43.060533 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.060377    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683"} err="failed to get container status \"01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683\": rpc error: code = NotFound desc = could not find container \"01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683\": container with ID starting with 01d63b417cd5bbe3274bb29863944d83d18a066425a757c40f0f0ee85df2c683 not found: ID does not exist"
Oct 31 08:24:43.060533 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.060395    4913 scope.go:122] "RemoveContainer" containerID="1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89"
Oct 31 08:24:43.060962 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:24:43.060944    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89\": container with ID starting with 1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89 not found: ID does not exist" containerID="1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89"
Oct 31 08:24:43.060962 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.060967    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89"} err="failed to get container status \"1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89\": rpc error: code = NotFound desc = could not find container \"1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89\": container with ID starting with 1ee97164df94e37a8263e8ad2f427d916f11b7ab3e1ac8c9f96f3d8d39018c89 not found: ID does not exist"
Oct 31 08:24:43.061298 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.061280    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-paddle-v2-kserve-predictor-686c49d88f-784sj"]
Oct 31 08:24:43.419428 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:43.419353    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" path="/var/lib/kubelet/pods/6b7521bc-ac6a-4159-b363-b2715425b93b/volumes"
Oct 31 08:24:45.010196 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010155    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"]
Oct 31 08:24:45.010719 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010688    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="storage-initializer"
Oct 31 08:24:45.010719 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010708    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="storage-initializer"
Oct 31 08:24:45.010852 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010737    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kube-rbac-proxy"
Oct 31 08:24:45.010852 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010747    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kube-rbac-proxy"
Oct 31 08:24:45.010852 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010777    4913 cpu_manager.go:414] "containerMap: removing container" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container"
Oct 31 08:24:45.010852 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010785    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container"
Oct 31 08:24:45.011041 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010875    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kube-rbac-proxy"
Oct 31 08:24:45.011041 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.010892    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7521bc-ac6a-4159-b363-b2715425b93b" containerName="kserve-container"
Oct 31 08:24:45.015917 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.015896    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.021566 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.021545    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:24:45.024300 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.024271    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"kube-root-ca.crt\""
Oct 31 08:24:45.025216 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.025170    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"seaweedfs-s3-creds\""
Oct 31 08:24:45.025515 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.025495    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"default-dockercfg-fzf82\""
Oct 31 08:24:45.025624 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.025574    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Oct 31 08:24:45.026912 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.026889    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"]
Oct 31 08:24:45.051995 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.051967    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c36d88-daa5-44f0-bb9c-b620795a6086-proxy-tls\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.052207 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.052001    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.052207 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.052020    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.052207 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.052037    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqgg9\" (UniqueName: \"kubernetes.io/projected/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-api-access-tqgg9\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.052207 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.052152    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c36d88-daa5-44f0-bb9c-b620795a6086-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.123597 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.123569    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"openshift-service-ca.crt\""
Oct 31 08:24:45.123597 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.123590    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-9226a10e02\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Oct 31 08:24:45.152948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.152920    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c36d88-daa5-44f0-bb9c-b620795a6086-proxy-tls\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.153028 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.152965    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.153028 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.152983    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.153111 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.153026    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqgg9\" (UniqueName: \"kubernetes.io/projected/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-api-access-tqgg9\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.153111 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.153065    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c36d88-daa5-44f0-bb9c-b620795a6086-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.153519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.153495    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c36d88-daa5-44f0-bb9c-b620795a6086-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.153756 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.153737    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.153832 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.153810    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.155551 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.155532    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c36d88-daa5-44f0-bb9c-b620795a6086-proxy-tls\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.160519 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.160492    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqgg9\" (UniqueName: \"kubernetes.io/projected/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-api-access-tqgg9\") pod \"isvc-pmml-runtime-predictor-5cdfb65444-7m2zl\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") " pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.334656 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.334566    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:45.606025 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:45.605950    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"]
Oct 31 08:24:45.609084 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:24:45.609058    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c36d88_daa5_44f0_bb9c_b620795a6086.slice/crio-1a8c3817bd981d0e9c177697d2cf66a7432a9676e2dbf6a46a7e1f6f5cf81094 WatchSource:0}: Error finding container 1a8c3817bd981d0e9c177697d2cf66a7432a9676e2dbf6a46a7e1f6f5cf81094: Status 404 returned error can't find the container with id 1a8c3817bd981d0e9c177697d2cf66a7432a9676e2dbf6a46a7e1f6f5cf81094
Oct 31 08:24:46.052534 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:46.052500    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" event={"ID":"39c36d88-daa5-44f0-bb9c-b620795a6086","Type":"ContainerStarted","Data":"70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93"}
Oct 31 08:24:46.052919 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:46.052559    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" event={"ID":"39c36d88-daa5-44f0-bb9c-b620795a6086","Type":"ContainerStarted","Data":"1a8c3817bd981d0e9c177697d2cf66a7432a9676e2dbf6a46a7e1f6f5cf81094"}
Oct 31 08:24:49.400329 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:24:49.400289    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:24:50.070612 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:50.070576    4913 generic.go:356] "Generic (PLEG): container finished" podID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerID="70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93" exitCode=0
Oct 31 08:24:50.070770 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:50.070648    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" event={"ID":"39c36d88-daa5-44f0-bb9c-b620795a6086","Type":"ContainerDied","Data":"70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93"}
Oct 31 08:24:58.112426 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:58.112390    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" event={"ID":"39c36d88-daa5-44f0-bb9c-b620795a6086","Type":"ContainerStarted","Data":"12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d"}
Oct 31 08:24:58.112855 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:58.112451    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" event={"ID":"39c36d88-daa5-44f0-bb9c-b620795a6086","Type":"ContainerStarted","Data":"adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752"}
Oct 31 08:24:58.112855 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:58.112695    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:58.131850 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:58.131800    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podStartSLOduration=2.723742279 podStartE2EDuration="14.131785431s" totalImagesPullingTime="7.466520335s" totalInitContainerRuntime="3.941522817s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:24:58.131070597 +0000 UTC m=+3346.820838901" watchObservedRunningTime="2026-10-31 08:24:58.131785431 +0000 UTC m=+3346.821553722"
Oct 31 08:24:59.116484 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:59.116452    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:24:59.118034 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:24:59.118006    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:25:00.120744 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:25:00.120708    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:25:05.125889 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:25:05.125851    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:25:05.126514 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:25:05.126485    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:25:15.127421 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:25:15.127379    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:25:19.399698 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:25:19.399657    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:25:25.127028 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:25:25.126981    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:25:35.126921 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:25:35.126870    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:25:45.126454 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:25:45.126409    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:25:49.399640 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:25:49.399602    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:25:55.127072 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:25:55.127023    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:26:05.127021 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:05.126978    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:26:15.127220 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:15.127158    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:26:19.400267 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:26:19.400224    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:26:24.414083 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:24.414043    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.87:8080: connect: connection refused"
Oct 31 08:26:34.414860 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:34.414829    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:26:36.954115 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:36.954080    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"]
Oct 31 08:26:36.954818 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:36.954362    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container" containerID="cri-o://adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752" gracePeriod=30
Oct 31 08:26:36.954818 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:36.954408    4913 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kube-rbac-proxy" containerID="cri-o://12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d" gracePeriod=30
Oct 31 08:26:37.512912 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:37.512876    4913 generic.go:356] "Generic (PLEG): container finished" podID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerID="12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d" exitCode=0
Oct 31 08:26:37.513095 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:37.512944    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" event={"ID":"39c36d88-daa5-44f0-bb9c-b620795a6086","Type":"ContainerDied","Data":"12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d"}
Oct 31 08:26:40.121225 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:40.121162    4913 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.87:8643/healthz\": dial tcp 10.132.0.87:8643: connect: connection refused"
Oct 31 08:26:41.200125 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.200099    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:26:41.345510 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.345422    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-rbac-proxy-sar-config\") pod \"39c36d88-daa5-44f0-bb9c-b620795a6086\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") "
Oct 31 08:26:41.345510 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.345462    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-cabundle-cert\") pod \"39c36d88-daa5-44f0-bb9c-b620795a6086\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") "
Oct 31 08:26:41.345510 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.345507    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c36d88-daa5-44f0-bb9c-b620795a6086-proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c36d88-daa5-44f0-bb9c-b620795a6086-proxy-tls\") pod \"39c36d88-daa5-44f0-bb9c-b620795a6086\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") "
Oct 31 08:26:41.345797 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.345626    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c36d88-daa5-44f0-bb9c-b620795a6086-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c36d88-daa5-44f0-bb9c-b620795a6086-kserve-provision-location\") pod \"39c36d88-daa5-44f0-bb9c-b620795a6086\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") "
Oct 31 08:26:41.345797 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.345683    4913 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-api-access-tqgg9\" (UniqueName: \"kubernetes.io/projected/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-api-access-tqgg9\") pod \"39c36d88-daa5-44f0-bb9c-b620795a6086\" (UID: \"39c36d88-daa5-44f0-bb9c-b620795a6086\") "
Oct 31 08:26:41.345931 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.345908    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-cabundle-cert" pod "39c36d88-daa5-44f0-bb9c-b620795a6086" (UID: "39c36d88-daa5-44f0-bb9c-b620795a6086"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:26:41.345986 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.345898    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-rbac-proxy-sar-config" pod "39c36d88-daa5-44f0-bb9c-b620795a6086" (UID: "39c36d88-daa5-44f0-bb9c-b620795a6086"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:26:41.345986 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.345935    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c36d88-daa5-44f0-bb9c-b620795a6086-kserve-provision-location" pod "39c36d88-daa5-44f0-bb9c-b620795a6086" (UID: "39c36d88-daa5-44f0-bb9c-b620795a6086"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:26:41.347722 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.347700    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-api-access-tqgg9" pod "39c36d88-daa5-44f0-bb9c-b620795a6086" (UID: "39c36d88-daa5-44f0-bb9c-b620795a6086"). InnerVolumeSpecName "kube-api-access-tqgg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:26:41.347722 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.347715    4913 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c36d88-daa5-44f0-bb9c-b620795a6086-proxy-tls" pod "39c36d88-daa5-44f0-bb9c-b620795a6086" (UID: "39c36d88-daa5-44f0-bb9c-b620795a6086"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:26:41.446282 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.446253    4913 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c36d88-daa5-44f0-bb9c-b620795a6086-kserve-provision-location\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:26:41.446282 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.446278    4913 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqgg9\" (UniqueName: \"kubernetes.io/projected/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-api-access-tqgg9\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:26:41.446457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.446287    4913 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:26:41.446457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.446297    4913 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c36d88-daa5-44f0-bb9c-b620795a6086-cabundle-cert\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:26:41.446457 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.446305    4913 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c36d88-daa5-44f0-bb9c-b620795a6086-proxy-tls\") on node \"ip-10-0-130-121.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 08:26:41.530762 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.530728    4913 generic.go:356] "Generic (PLEG): container finished" podID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerID="adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752" exitCode=0
Oct 31 08:26:41.530948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.530798    4913 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"
Oct 31 08:26:41.530948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.530816    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" event={"ID":"39c36d88-daa5-44f0-bb9c-b620795a6086","Type":"ContainerDied","Data":"adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752"}
Oct 31 08:26:41.530948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.530892    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl" event={"ID":"39c36d88-daa5-44f0-bb9c-b620795a6086","Type":"ContainerDied","Data":"1a8c3817bd981d0e9c177697d2cf66a7432a9676e2dbf6a46a7e1f6f5cf81094"}
Oct 31 08:26:41.530948 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.530928    4913 scope.go:122] "RemoveContainer" containerID="12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d"
Oct 31 08:26:41.538928 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.538911    4913 scope.go:122] "RemoveContainer" containerID="adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752"
Oct 31 08:26:41.546083 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.546068    4913 scope.go:122] "RemoveContainer" containerID="70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93"
Oct 31 08:26:41.548607 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.548584    4913 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"]
Oct 31 08:26:41.553109 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.553087    4913 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-9226a10e02/isvc-pmml-runtime-predictor-5cdfb65444-7m2zl"]
Oct 31 08:26:41.554126 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.554111    4913 scope.go:122] "RemoveContainer" containerID="12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d"
Oct 31 08:26:41.554405 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:26:41.554386    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d\": container with ID starting with 12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d not found: ID does not exist" containerID="12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d"
Oct 31 08:26:41.554473 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.554413    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d"} err="failed to get container status \"12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d\": rpc error: code = NotFound desc = could not find container \"12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d\": container with ID starting with 12917e8d5a3c2c020e03006bba46e90641b468205f6d3202572e90d8d58b552d not found: ID does not exist"
Oct 31 08:26:41.554473 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.554431    4913 scope.go:122] "RemoveContainer" containerID="adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752"
Oct 31 08:26:41.554688 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:26:41.554669    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752\": container with ID starting with adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752 not found: ID does not exist" containerID="adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752"
Oct 31 08:26:41.554756 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.554697    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752"} err="failed to get container status \"adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752\": rpc error: code = NotFound desc = could not find container \"adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752\": container with ID starting with adaeeaa66b117ebdd90846f3d0f13d1425e1120dd359be8f33f7a037e36e9752 not found: ID does not exist"
Oct 31 08:26:41.554756 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.554719    4913 scope.go:122] "RemoveContainer" containerID="70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93"
Oct 31 08:26:41.554959 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:26:41.554943    4913 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93\": container with ID starting with 70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93 not found: ID does not exist" containerID="70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93"
Oct 31 08:26:41.555016 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:41.554966    4913 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93"} err="failed to get container status \"70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93\": rpc error: code = NotFound desc = could not find container \"70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93\": container with ID starting with 70b651e8789229d9b6c18c38c5553f586becd6ba581ad0df6a1b18b57aa4df93 not found: ID does not exist"
Oct 31 08:26:43.419243 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:26:43.419213    4913 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" path="/var/lib/kubelet/pods/39c36d88-daa5-44f0-bb9c-b620795a6086/volumes"
Oct 31 08:26:49.399057 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:26:49.399014    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:27:19.400339 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:27:19.400302    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:27:49.399533 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:27:49.399489    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:28:02.313246 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:02.313212    4913 ???:1] "http: TLS handshake error from 10.0.137.69:54918: EOF"
Oct 31 08:28:02.319410 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:02.319382    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l4tmf_9420864f-c0f7-47d3-ad36-b4b15ba44b57/global-pull-secret-syncer/0.log"
Oct 31 08:28:02.406028 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:02.405995    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-22c6v_385e4d20-8bd7-4933-8a7d-4359e0301d89/connection-checker/0.log"
Oct 31 08:28:02.430523 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:02.430501    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5nzmp_1820478c-a566-4c78-b063-e8ceb9c6e221/connection-checker/0.log"
Oct 31 08:28:02.456109 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:02.456084    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ch85h_0d264640-a684-42bd-8787-86d2382a7eb4/connection-checker/0.log"
Oct 31 08:28:02.532152 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:02.532120    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sbln7_6121aa17-a37d-47de-af76-08c2aa174568/konnectivity-agent/0.log"
Oct 31 08:28:02.624616 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:02.624545    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-121.us-east-2.compute.internal_a42df5a3903764c9a9afa19fe2c78097/haproxy/0.log"
Oct 31 08:28:07.855025 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:07.854992    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-qrsxv_8b210005-d749-4ec8-ad41-bc7afaeb7011/cluster-monitoring-operator/0.log"
Oct 31 08:28:08.039535 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:08.039493    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9kkbv_c91e4782-7ff8-4129-87cc-df5e71fed629/node-exporter/0.log"
Oct 31 08:28:08.065315 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:08.065285    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9kkbv_c91e4782-7ff8-4129-87cc-df5e71fed629/kube-rbac-proxy/0.log"
Oct 31 08:28:08.093448 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:08.093425    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9kkbv_c91e4782-7ff8-4129-87cc-df5e71fed629/init-textfile/0.log"
Oct 31 08:28:10.004707 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.004679    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-jgczn_c82b956f-d109-4a4e-9a90-f298f5dcaf11/networking-console-plugin/0.log"
Oct 31 08:28:10.316374 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316302    4913 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"]
Oct 31 08:28:10.316620 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316609    4913 cpu_manager.go:414] "containerMap: removing container" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container"
Oct 31 08:28:10.316667 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316622    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container"
Oct 31 08:28:10.316667 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316636    4913 cpu_manager.go:414] "containerMap: removing container" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kube-rbac-proxy"
Oct 31 08:28:10.316667 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316641    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kube-rbac-proxy"
Oct 31 08:28:10.316667 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316654    4913 cpu_manager.go:414] "containerMap: removing container" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="storage-initializer"
Oct 31 08:28:10.316667 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316659    4913 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="storage-initializer"
Oct 31 08:28:10.316818 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316743    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kube-rbac-proxy"
Oct 31 08:28:10.316818 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.316758    4913 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c36d88-daa5-44f0-bb9c-b620795a6086" containerName="kserve-container"
Oct 31 08:28:10.319880 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.319861    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.329027 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.329005    4913 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jbks9\"/\"default-dockercfg-dpgr6\""
Oct 31 08:28:10.329468 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.329450    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jbks9\"/\"kube-root-ca.crt\""
Oct 31 08:28:10.329600 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.329480    4913 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jbks9\"/\"openshift-service-ca.crt\""
Oct 31 08:28:10.330198 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.330163    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"]
Oct 31 08:28:10.441046 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.441004    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-proc\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.441286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.441065    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cg2\" (UniqueName: \"kubernetes.io/projected/b42795f3-d90d-49ec-9629-736d76c987e0-kube-api-access-78cg2\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.441286 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.441229    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-podres\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.441395 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.441285    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-sys\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.441395 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.441346    4913 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-lib-modules\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542250 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542209    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cg2\" (UniqueName: \"kubernetes.io/projected/b42795f3-d90d-49ec-9629-736d76c987e0-kube-api-access-78cg2\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542265    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-podres\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542299    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-sys\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542324    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-lib-modules\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542367    4913 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-proc\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542398    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-podres\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542404    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-sys\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542445 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542433    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-lib-modules\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.542696 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.542451    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b42795f3-d90d-49ec-9629-736d76c987e0-proc\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.550109 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.550076    4913 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cg2\" (UniqueName: \"kubernetes.io/projected/b42795f3-d90d-49ec-9629-736d76c987e0-kube-api-access-78cg2\") pod \"perf-node-gather-daemonset-rpsrp\" (UID: \"b42795f3-d90d-49ec-9629-736d76c987e0\") " pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.591489 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.591417    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/2.log"
Oct 31 08:28:10.596081 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.596057    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7h5mf_9a1a70f5-55db-4e36-959d-95bae0872fea/console-operator/3.log"
Oct 31 08:28:10.633010 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.632982    4913 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:10.902745 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:10.902654    4913 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"]
Oct 31 08:28:10.905554 ip-10-0-130-121 kubenswrapper[4913]: W1031 08:28:10.905517    4913 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb42795f3_d90d_49ec_9629_736d76c987e0.slice/crio-a01d31190854350a063f87ff92b855f70c208f49e8857f7836e6d986df0fc86f WatchSource:0}: Error finding container a01d31190854350a063f87ff92b855f70c208f49e8857f7836e6d986df0fc86f: Status 404 returned error can't find the container with id a01d31190854350a063f87ff92b855f70c208f49e8857f7836e6d986df0fc86f
Oct 31 08:28:11.612655 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:11.612622    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-t6dgh_83e1b5fd-c06c-45a2-9c74-f29515e920ff/volume-data-source-validator/0.log"
Oct 31 08:28:11.895730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:11.895647    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp" event={"ID":"b42795f3-d90d-49ec-9629-736d76c987e0","Type":"ContainerStarted","Data":"66d1e9b1a618d4f22f888f92b77ba473558d21bd3a860c12b46aa881dafceba5"}
Oct 31 08:28:11.895730 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:11.895698    4913 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp" event={"ID":"b42795f3-d90d-49ec-9629-736d76c987e0","Type":"ContainerStarted","Data":"a01d31190854350a063f87ff92b855f70c208f49e8857f7836e6d986df0fc86f"}
Oct 31 08:28:11.895941 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:11.895761    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:11.911649 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:11.911594    4913 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp" podStartSLOduration=1.9115809129999999 podStartE2EDuration="1.911580913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 08:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:28:11.910300605 +0000 UTC m=+3540.600068909" watchObservedRunningTime="2026-10-31 08:28:11.911580913 +0000 UTC m=+3540.601349204"
Oct 31 08:28:12.600431 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:12.600407    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46cpb_3adfe0f6-05ab-44f9-8884-06c021c42f5a/dns/0.log"
Oct 31 08:28:12.628765 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:12.628736    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46cpb_3adfe0f6-05ab-44f9-8884-06c021c42f5a/kube-rbac-proxy/0.log"
Oct 31 08:28:12.789148 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:12.789124    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-49r2r_5b3d13df-dc50-41c8-a7cd-051fb9934a34/dns-node-resolver/0.log"
Oct 31 08:28:13.349068 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:13.349037    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fz4tp_4acf6e61-938d-495e-92ea-c1bc343b9406/node-ca/0.log"
Oct 31 08:28:14.217353 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:14.217325    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6f75f655-wgvm2_bd4df54c-65e4-4dba-943c-17219493fc36/router/0.log"
Oct 31 08:28:14.601262 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:14.601173    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w9f8b_63139743-194a-4517-9eba-62a858a3b649/serve-healthcheck-canary/0.log"
Oct 31 08:28:15.023342 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:15.023313    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-rjjcf_e684ad1d-ea28-4d75-aca0-8c12227ffada/insights-operator/0.log"
Oct 31 08:28:15.024438 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:15.024417    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-rjjcf_e684ad1d-ea28-4d75-aca0-8c12227ffada/insights-operator/1.log"
Oct 31 08:28:15.048721 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:15.048694    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cdwzx_b0e85bb8-780b-4540-8e9e-19990f22f515/kube-rbac-proxy/0.log"
Oct 31 08:28:15.076993 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:15.076966    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cdwzx_b0e85bb8-780b-4540-8e9e-19990f22f515/exporter/0.log"
Oct 31 08:28:15.101811 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:15.101784    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cdwzx_b0e85bb8-780b-4540-8e9e-19990f22f515/extractor/0.log"
Oct 31 08:28:17.699200 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:17.699146    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-76d8b77fbb-mq67h_7c5269f9-4eef-43d7-a2a9-7d134c7e9bae/manager/0.log"
Oct 31 08:28:17.911485 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:17.911453    4913 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jbks9/perf-node-gather-daemonset-rpsrp"
Oct 31 08:28:18.380088 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:18.380057    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-685c9f986c-6qbvb_e407da07-e3c6-44f1-979a-fb525bf8c30b/seaweedfs-tls-custom/0.log"
Oct 31 08:28:18.406377 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:18.406346    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-5c74974785-nq7pq_a012adb2-0a1b-47d4-9eb7-115261749051/seaweedfs-tls-serving/0.log"
Oct 31 08:28:19.400853 ip-10-0-130-121 kubenswrapper[4913]: E1031 08:28:19.400812    4913 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cdwzx" containerName="extractor"
Oct 31 08:28:22.834957 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:22.834925    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-tbdww_12778afe-1cd7-4829-b2c5-ddf2f1232ac2/migrator/0.log"
Oct 31 08:28:23.162979 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:23.162880    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-l546l_87fbf516-95a9-4c2c-a41f-1a053ff98e6b/kube-storage-version-migrator-operator/1.log"
Oct 31 08:28:23.163712 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:23.163692    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-l546l_87fbf516-95a9-4c2c-a41f-1a053ff98e6b/kube-storage-version-migrator-operator/0.log"
Oct 31 08:28:24.060033 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.060005    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4l58_0546b914-d103-41a5-9e5b-13c6e41bcd85/kube-multus-additional-cni-plugins/0.log"
Oct 31 08:28:24.085076 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.085033    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4l58_0546b914-d103-41a5-9e5b-13c6e41bcd85/egress-router-binary-copy/0.log"
Oct 31 08:28:24.109571 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.109547    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4l58_0546b914-d103-41a5-9e5b-13c6e41bcd85/cni-plugins/0.log"
Oct 31 08:28:24.134462 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.134437    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4l58_0546b914-d103-41a5-9e5b-13c6e41bcd85/bond-cni-plugin/0.log"
Oct 31 08:28:24.159059 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.159031    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4l58_0546b914-d103-41a5-9e5b-13c6e41bcd85/routeoverride-cni/0.log"
Oct 31 08:28:24.182490 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.182465    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4l58_0546b914-d103-41a5-9e5b-13c6e41bcd85/whereabouts-cni-bincopy/0.log"
Oct 31 08:28:24.205321 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.205298    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4l58_0546b914-d103-41a5-9e5b-13c6e41bcd85/whereabouts-cni/0.log"
Oct 31 08:28:24.667108 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.667084    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hmj2s_5b8e87b5-3eda-4fd4-9b70-f72fe97084ca/kube-multus/0.log"
Oct 31 08:28:24.822579 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.822501    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-svhj9_b5bca13e-12d2-4673-8223-92f7755eac90/network-metrics-daemon/0.log"
Oct 31 08:28:24.843019 ip-10-0-130-121 kubenswrapper[4913]: I1031 08:28:24.842992    4913 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-svhj9_b5bca13e-12d2-4673-8223-92f7755eac90/kube-rbac-proxy/0.log"