Sep 17 15:53:52.359561 ip-10-0-142-18 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:53:55.568044 ip-10-0-142-18 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:53:55.568044 ip-10-0-142-18 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:53:55.568044 ip-10-0-142-18 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:53:55.568044 ip-10-0-142-18 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:53:55.568044 ip-10-0-142-18 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:53:55.568835 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.568132    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572163    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572183    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572186    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572190    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572193    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572196    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572199    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572201    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572204    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:53:55.572193 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572208    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572211    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572214    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572217    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572220    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572224    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572228    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572231    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572234    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572237    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572240    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572242    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572246    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572250    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572253    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572256    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572259    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572263    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:53:55.572562 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572267    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572273    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572276    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572295    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572300    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572304    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572307    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572309    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572312    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572315    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572318    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572322    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572325    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572327    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572330    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572333    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572336    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572339    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572341    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572344    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:53:55.573018 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572347    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572350    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572353    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572355    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572359    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572361    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572364    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572366    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572369    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572372    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572374    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572377    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572379    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572382    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572384    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572387    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572389    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572399    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572403    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572406    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:53:55.573545 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572408    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572411    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572413    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572416    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572418    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572421    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572424    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572426    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572430    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572433    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572436    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572439    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572442    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572445    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572449    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572452    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572456    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572458    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572461    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572464    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:53:55.574031 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572466    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572469    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572472    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572475    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572478    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572480    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572483    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572486    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572489    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572492    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572494    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.572497    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573156    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573162    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573165    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573168    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573171    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573174    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573177    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573180    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:53:55.574563 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573183    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573186    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573189    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573191    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573194    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573196    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573199    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573202    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573204    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573208    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573210    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573213    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573216    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573219    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573221    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573224    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573227    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573229    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573232    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:53:55.575086 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573235    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573238    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573240    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573243    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573246    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573248    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573251    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573253    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573256    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573259    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573262    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573264    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573267    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573270    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573273    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573276    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573278    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573301    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573305    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573307    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:53:55.575612 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573310    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573313    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573315    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573319    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573322    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573325    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573327    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573331    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573334    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573336    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573339    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573342    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573344    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573347    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573350    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573353    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573355    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573360    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573363    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:53:55.576147 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573366    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573369    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573372    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573375    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573378    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573381    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573384    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573389    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573393    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573397    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573400    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573403    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573406    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573409    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573411    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573414    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573417    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573420    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573423    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573426    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:53:55.576638 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573428    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573431    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573434    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573437    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573440    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573442    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573445    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573447    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573450    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573452    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573455    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573458    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573460    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573830    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573835    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573838    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573840    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573843    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573846    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573849    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:53:55.577141 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573852    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573855    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573857    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573860    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573863    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573866    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573868    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573871    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573874    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573877    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573880    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573882    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573885    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573888    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573890    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573893    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573896    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573899    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573902    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573905    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:53:55.577637 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573908    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573910    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573913    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573915    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573918    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573921    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573924    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573927    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573929    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573932    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573935    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573937    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573940    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573943    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573946    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573948    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573951    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573953    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573956    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573959    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:53:55.578151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573961    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573966    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573970    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573973    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573976    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573979    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573982    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573986    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573991    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573994    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573996    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.573999    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574002    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574004    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574007    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574011    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574015    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574018    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574021    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:53:55.578672 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574024    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574027    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574032    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574036    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574040    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574042    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574046    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574048    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574051    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574054    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574056    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574059    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574061    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574064    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574067    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574069    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574073    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574076    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574079    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:53:55.579131 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574081    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574084    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574088    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574090    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574093    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574095    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574098    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574101    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574103    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574106    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574109    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574111    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574114    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574116    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574191    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574226    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574232    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574237    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574241    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574244    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574248    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:53:55.579610 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574252    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574255    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574258    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574262    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574265    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574268    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574271    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574274    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574276    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574297    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574302    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574306    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574311    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574314    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574318    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574321    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574325    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574328    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574331    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574335    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574338    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574341    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574344    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574347    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:53:55.580112 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574350    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574354    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574357    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574360    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574363    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574366    2585 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574369    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574374    2585 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574377    2585 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574380    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574383    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574386    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574389    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574392    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574395    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574398    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574401    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574405    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574409    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574412    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574416    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574419    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574422    2585 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574426    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574429    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:53:55.580711 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574432    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574435    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574438    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574441    2585 flags.go:64] FLAG: --help="false"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574444    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574448    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574451    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574454    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574457    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574460    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574463    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574466    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574469    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574472    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574474    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574477    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574480    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574483    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574487    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574490    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574492    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574495    2585 flags.go:64] FLAG: --lock-file=""
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574498    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574501    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:53:55.581465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574504    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574509    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574513    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574517    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574520    2585 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574523    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574526    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574529    2585 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574532    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574536    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574539    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574543    2585 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574546    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574549    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574552    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574554    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574557    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574560    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574563    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574570    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574573    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574577    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574580    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:53:55.582052 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574582    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574588    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574591    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574594    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574597    2585 flags.go:64] FLAG: --port="10250"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574600    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574603    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b0ee8ddf1d71c581"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574606    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574609    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574612    2585 flags.go:64] FLAG: --register-node="true"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574615    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574618    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574624    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574627    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574631    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574635    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574638    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574641    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574644    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574647    2585 flags.go:64] FLAG: --runonce="false"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574650    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574653    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574656    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574659    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574662    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574665    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:53:55.582667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574668    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574671    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574674    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574677    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574680    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574683    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574686    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574689    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574692    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574698    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574700    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574703    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574707    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574710    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574713    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574715    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574718    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574721    2585 flags.go:64] FLAG: --v="2"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574726    2585 flags.go:64] FLAG: --version="false"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574731    2585 flags.go:64] FLAG: --vmodule=""
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574736    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.574739    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574849    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574853    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:53:55.583318 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574856    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574860    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574863    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574866    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574869    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574873    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574878    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574881    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574884    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574887    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574889    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574892    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574896    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574898    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574901    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574904    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574907    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574909    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:53:55.583907 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574912    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574915    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574918    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574920    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574923    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574925    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574928    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574931    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574935    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574938    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574943    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574946    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574948    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574951    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574954    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574956    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574959    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574962    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574965    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574967    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:53:55.584433 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574970    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574972    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574975    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574978    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574981    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574983    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574986    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574989    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574991    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574994    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574997    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.574999    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575002    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575005    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575007    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575011    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575013    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575016    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575019    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575021    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:53:55.584920 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575025    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575028    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575031    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575035    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575038    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575041    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575043    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575046    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575049    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575051    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575054    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575057    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575060    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575062    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575065    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575068    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575071    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575073    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575076    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575079    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:53:55.585438 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575081    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575084    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575087    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575089    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575093    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575095    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575098    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575101    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575103    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575106    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575109    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575112    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575115    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575118    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575123    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575127    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575131    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575134    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:53:55.585936 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.575137    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.575699    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.582400    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.582416    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.582432    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.582438    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582493    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582498    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582502    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582505    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582508    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582512    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582515    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582518    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582521    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:53:55.586398 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582524    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582528    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582531    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582533    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582537    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582540    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582542    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582545    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582548    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582551    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582554    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582557    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582560    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582562    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582565    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582568    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582570    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582575    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582579    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:53:55.586778 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582583    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582588    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582592    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582595    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582598    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582601    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582604    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582607    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582610    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582614    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582618    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582621    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582624    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582627    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582630    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582633    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582635    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582638    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582641    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582644    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:53:55.587239 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582646    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582649    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582652    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582654    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582657    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582660    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582662    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582665    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582668    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582670    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582673    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582676    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582679    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582683    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582686    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582689    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582692    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582694    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582697    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582700    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:53:55.587746 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582702    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582705    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582707    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582710    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582713    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582715    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582718    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582720    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582723    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582725    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582728    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582731    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582733    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582736    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582739    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582741    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582744    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582747    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582749    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:53:55.588264 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582752    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582755    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582757    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582760    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582762    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582765    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582769    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582771    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582775    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582777    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582780    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582782    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.582787    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582893    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582898    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:53:55.588826 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582902    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582905    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582908    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582911    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582913    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582916    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582918    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582921    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582924    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582927    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582929    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582932    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582934    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582937    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582940    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582942    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582945    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582948    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582950    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582953    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:53:55.589197 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582955    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582958    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582960    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582963    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582966    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582970    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582974    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582977    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582980    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582983    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582986    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582989    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582991    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582994    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582997    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.582999    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583002    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583004    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583007    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:53:55.589690 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583010    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583012    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583015    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583018    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583020    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583023    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583025    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583028    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583031    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583033    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583036    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583038    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583041    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583044    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583046    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583049    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583052    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583054    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583057    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583060    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:53:55.590156 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583062    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583065    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583068    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583070    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583073    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583075    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583078    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583081    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583083    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583086    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583088    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583091    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583094    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583097    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583099    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583102    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583104    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583107    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583110    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583114    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:53:55.590651 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583117    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583119    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583122    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583125    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583127    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583130    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583133    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583136    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583139    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583141    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583144    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583147    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583149    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583152    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583155    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583157    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583160    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:53:55.591301 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:55.583163    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:53:55.591816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.583167    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:53:55.591816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.583258    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:53:55.591816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.585297    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:53:55.591816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.586060    2585 server.go:1019] "Starting client certificate rotation"
Sep 17 15:53:55.591816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.586158    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:53:55.591816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.586190    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:53:55.591816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.588749    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:53:55.591816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.590705    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:53:55.603356 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.603339    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:53:55.606517 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.606497    2585 log.go:25] "Validated CRI v1 image API"
Sep 17 15:53:55.607340 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.607323    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:53:55.609316 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.609275    2585 fs.go:135] Filesystem UUIDs: map[49e5c99d-7473-4c04-81cf-4b7e2fc0b271:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b0b9dc24-223d-4f21-94f1-7921aad9aba0:/dev/nvme0n1p3]
Sep 17 15:53:55.609378 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.609314    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:53:55.613822 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.613712    2585 manager.go:216] Machine: {Timestamp:2026-09-17 15:53:55.613235568 +0000 UTC m=+1.878750558 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109825 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c2c2bce7469bdcbfb89f25276f6ad SystemUUID:ec2c2c2b-ce74-69bd-cbfb-89f25276f6ad BootID:20bace0c-7c34-4078-9813-2915d50aa99c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:a4:0a:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:a4:0a:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:c2:99:47:67:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:53:55.613822 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.613814    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:53:55.613925 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.613883    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:53:55.615387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.615365    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:53:55.615526 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.615390    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-18.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:53:55.616524 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.616514    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:53:55.616558 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.616527    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:53:55.616558 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.616535    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:53:55.616558 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.616546    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:53:55.616641 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.616611    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:53:55.620228 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620218    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:53:55.620355 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620345    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:53:55.620386 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620378    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:53:55.620427 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620387    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:53:55.620427 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620407    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:53:55.620427 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620416    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:53:55.620427 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620424    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:53:55.620751 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620738    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:53:55.620807 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.620758    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:53:55.622608 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.622590    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 15:53:55.625028 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.625014    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:53:55.625071 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.625045    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:53:55.628836 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628819    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:53:55.628900 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628846    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:53:55.628900 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628854    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:53:55.628900 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628860    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:53:55.628900 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628872    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:53:55.628900 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628879    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:53:55.628900 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628898    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:53:55.629054 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628904    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:53:55.629054 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628914    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:53:55.629054 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628919    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:53:55.629054 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628928    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:53:55.629054 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628933    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:53:55.629054 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628959    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:53:55.629054 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.628966    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:53:55.632628 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.632611    2585 server.go:1268] "Started kubelet"
Sep 17 15:53:55.632706 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.632667    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:53:55.632787 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.632763    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:53:55.632903 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.632869    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:53:55.633371 ip-10-0-142-18 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:53:55.633876 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.633859    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:53:55.665528 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.665504    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:53:55.665736 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.665694    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:53:55.665840 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.665685    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:53:55.665840 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.665804    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-18.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:53:55.665980 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.665898    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:53:55.666717 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.666699    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:53:55.670727 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.669948    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-18.ec2.internal.18d626af2671435c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-18.ec2.internal,UID:ip-10-0-142-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-18.ec2.internal,},FirstTimestamp:2026-09-17 15:53:55.632440156 +0000 UTC m=+1.897955147,LastTimestamp:2026-09-17 15:53:55.632440156 +0000 UTC m=+1.897955147,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-18.ec2.internal,}"
Sep 17 15:53:55.671959 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.671945    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:53:55.671959 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.671953    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:53:55.672081 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.672018    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:55.672629 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.672607    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:53:55.672629 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.672608    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:53:55.672769 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.672635    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:53:55.672769 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.672724    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:53:55.672769 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.672734    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:53:55.672900 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.672773    2585 factory.go:55] Registering systemd factory
Sep 17 15:53:55.672900 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.672789    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:53:55.673748 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.673733    2585 factory.go:153] Registering CRI-O factory
Sep 17 15:53:55.673748 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.673749    2585 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:53:55.673877 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.673795    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:53:55.673877 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.673810    2585 factory.go:103] Registering Raw factory
Sep 17 15:53:55.673877 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.673821    2585 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:53:55.674278 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.674266    2585 manager.go:349] Starting recovery of all containers
Sep 17 15:53:55.679765 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.679715    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:53:55.684586 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.684570    2585 manager.go:354] Recovery completed
Sep 17 15:53:55.686572 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.686548    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:53:55.686787 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.686766    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:53:55.689143 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.689131    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:53:55.690781 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.690758    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:53:55.690846 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.690783    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:53:55.690846 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.690793    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:53:55.691422 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.691412    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:53:55.691459 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.691423    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:53:55.691459 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.691437    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:53:55.692732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.692712    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:53:55.693437 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.693420    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:53:55.693521 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.693453    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:53:55.693521 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.693471    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:53:55.693730 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.693718    2585 policy_none.go:49] "None policy: Start"
Sep 17 15:53:55.693775 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.693733    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:53:55.693775 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.693744    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:53:55.695683 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.695671    2585 policy_none.go:47] "Start"
Sep 17 15:53:55.705779 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.705712    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-18.ec2.internal.18d626af29eb6073  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-18.ec2.internal,UID:ip-10-0-142-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-18.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-18.ec2.internal,},FirstTimestamp:2026-09-17 15:53:55.690774643 +0000 UTC m=+1.956289634,LastTimestamp:2026-09-17 15:53:55.690774643 +0000 UTC m=+1.956289634,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-18.ec2.internal,}"
Sep 17 15:53:55.711206 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.711147    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-18.ec2.internal.18d626af29eb93fa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-18.ec2.internal,UID:ip-10-0-142-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-18.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-18.ec2.internal,},FirstTimestamp:2026-09-17 15:53:55.690787834 +0000 UTC m=+1.956302825,LastTimestamp:2026-09-17 15:53:55.690787834 +0000 UTC m=+1.956302825,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-18.ec2.internal,}"
Sep 17 15:53:55.720626 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.720561    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-18.ec2.internal.18d626af29ebb8df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-18.ec2.internal,UID:ip-10-0-142-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-18.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-18.ec2.internal,},FirstTimestamp:2026-09-17 15:53:55.690797279 +0000 UTC m=+1.956312270,LastTimestamp:2026-09-17 15:53:55.690797279 +0000 UTC m=+1.956312270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-18.ec2.internal,}"
Sep 17 15:53:55.731985 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.731967    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h75fs"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.733572    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-18.ec2.internal.18d626af29eb6073\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-18.ec2.internal.18d626af29eb6073  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-18.ec2.internal,UID:ip-10-0-142-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-18.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-18.ec2.internal,},FirstTimestamp:2026-09-17 15:53:55.690774643 +0000 UTC m=+1.956289634,LastTimestamp:2026-09-17 15:53:55.693435856 +0000 UTC m=+1.958950849,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-18.ec2.internal,}"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.736872    2585 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.736902    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.736912    2585 server.go:87] "Starting device plugin registration server"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.737125    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.737148    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.737308    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.737392    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.737401    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.737791    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.737843    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.745493    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-18.ec2.internal.18d626af29eb93fa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-18.ec2.internal.18d626af29eb93fa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-18.ec2.internal,UID:ip-10-0-142-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-18.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-18.ec2.internal,},FirstTimestamp:2026-09-17 15:53:55.690787834 +0000 UTC m=+1.956302825,LastTimestamp:2026-09-17 15:53:55.693461073 +0000 UTC m=+1.958976071,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-18.ec2.internal,}"
Sep 17 15:53:55.752190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.745719    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h75fs"
Sep 17 15:53:55.776447 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.776423    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:53:55.776523 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.776470    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:53:55.776523 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.776490    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:53:55.776588 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.776523    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:53:55.780119 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.780100    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:53:55.837261 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.837214    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:53:55.838041 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.838021    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:53:55.838121 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.838048    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:53:55.838121 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.838059    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:53:55.838121 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.838078    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.850067 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.850044    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.850153 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.850068    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-18.ec2.internal\": node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:55.877147 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.877126    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"]
Sep 17 15:53:55.877232 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.877181    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:53:55.878387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.878365    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:53:55.878475 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.878396    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:53:55.878475 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.878411    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:53:55.880494 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.880480    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:53:55.880617 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.880604    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.880663 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.880630    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:53:55.881143 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.881127    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:53:55.881199 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.881152    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:53:55.881199 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.881162    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:53:55.881257 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.881131    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:53:55.881257 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.881232    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:53:55.881257 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.881245    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:53:55.883348 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.883334    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.883412 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.883358    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:53:55.884013 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.883998    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:53:55.884079 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.884030    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:53:55.884079 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.884046    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:53:55.904927 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.904903    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-18.ec2.internal\" not found" node="ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.909238 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.909223    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-18.ec2.internal\" not found" node="ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.961799 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:55.961780    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:55.974972 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.974953    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acce870b27226e32c83f4f167afc6e1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"acce870b27226e32c83f4f167afc6e1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.975042 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.974975    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bca85661d4e2826a71d7270087fe9ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-18.ec2.internal\" (UID: \"3bca85661d4e2826a71d7270087fe9ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:55.975042 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:55.974990    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acce870b27226e32c83f4f167afc6e1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"acce870b27226e32c83f4f167afc6e1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.062212 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.062188    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.075596 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.075572    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bca85661d4e2826a71d7270087fe9ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-18.ec2.internal\" (UID: \"3bca85661d4e2826a71d7270087fe9ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.075696 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.075620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acce870b27226e32c83f4f167afc6e1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"acce870b27226e32c83f4f167afc6e1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.075696 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.075647    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acce870b27226e32c83f4f167afc6e1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"acce870b27226e32c83f4f167afc6e1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.075696 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.075681    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bca85661d4e2826a71d7270087fe9ca-config\") pod \"kube-apiserver-proxy-ip-10-0-142-18.ec2.internal\" (UID: \"3bca85661d4e2826a71d7270087fe9ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.075696 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.075687    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acce870b27226e32c83f4f167afc6e1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"acce870b27226e32c83f4f167afc6e1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.075696 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.075694    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acce870b27226e32c83f4f167afc6e1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal\" (UID: \"acce870b27226e32c83f4f167afc6e1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.163201 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.163123    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.208697 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.208662    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.212734 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.212708    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.263964 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.263809    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.364367 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.364332    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.464948 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.464885    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.565374 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.565345    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.585884 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.585860    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:53:56.586234 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.586022    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:53:56.665434 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.665408    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.676915 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.676894    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:53:56.752585 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.752546    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:48:55 +0000 UTC" deadline="2028-05-24 02:41:02.962618196 +0000 UTC"
Sep 17 15:53:56.752585 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.752582    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14746h47m6.210039157s"
Sep 17 15:53:56.756649 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.756633    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:53:56.766192 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.766176    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.781701 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.781680    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:53:56.782727 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.782713    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:53:56.782775 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.782741    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:53:56.782775 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.782753    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:53:56.867024 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:56.867000    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-18.ec2.internal\" not found"
Sep 17 15:53:56.931810 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.931783    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:53:56.972652 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.972631    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:56.989653 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.989637    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:53:56.990362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:56.990349    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal"
Sep 17 15:53:57.014384 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.014346    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:53:57.057547 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:57.057516    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacce870b27226e32c83f4f167afc6e1c.slice/crio-d850e34987bbaa22a69d395e8a8668ca8b0f902f2bde158c7beb50b8a7e26623 WatchSource:0}: Error finding container d850e34987bbaa22a69d395e8a8668ca8b0f902f2bde158c7beb50b8a7e26623: Status 404 returned error can't find the container with id d850e34987bbaa22a69d395e8a8668ca8b0f902f2bde158c7beb50b8a7e26623
Sep 17 15:53:57.057974 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:57.057949    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bca85661d4e2826a71d7270087fe9ca.slice/crio-dd6918da1be15d7f02947f7107a1f9552545d602db8a3ed4b6acfb9791d97e21 WatchSource:0}: Error finding container dd6918da1be15d7f02947f7107a1f9552545d602db8a3ed4b6acfb9791d97e21: Status 404 returned error can't find the container with id dd6918da1be15d7f02947f7107a1f9552545d602db8a3ed4b6acfb9791d97e21
Sep 17 15:53:57.062760 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.062682    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:53:57.621955 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.621923    2585 apiserver.go:52] "Watching apiserver"
Sep 17 15:53:57.631138 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.631112    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:53:57.631881 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.631851    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-s6qlb","openshift-dns/node-resolver-ksfvw","openshift-image-registry/node-ca-prh25","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal","openshift-network-operator/iptables-alerter-ht2p6","openshift-ovn-kubernetes/ovnkube-node-9ttzj","kube-system/global-pull-secret-syncer-q4f6r","kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj","openshift-cluster-node-tuning-operator/tuned-nxskj","openshift-multus/multus-additional-cni-plugins-s28wz","openshift-multus/multus-x5njn","openshift-multus/network-metrics-daemon-xh8xc","openshift-network-diagnostics/network-check-target-29lpf"]
Sep 17 15:53:57.634803 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.634779    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.637236 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.636986    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:57.637236 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.637069    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:53:57.639122 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.639101    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.639754 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.639731    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:53:57.639831 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.639809    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mhnzn\""
Sep 17 15:53:57.639831 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.639822    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:53:57.639924 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.639912    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:53:57.641599 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.641582    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.643585 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.643562    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.644375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.644354    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:53:57.644476 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.644391    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:53:57.644537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.644390    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:53:57.644537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.644488    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:53:57.645810 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.645793    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.645899 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.645862    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:53:57.645989 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.645971    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hxn67\""
Sep 17 15:53:57.646233 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.646217    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:53:57.646392 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.646378    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:53:57.646465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.646454    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:53:57.646620 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.646599    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:53:57.646678 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.646627    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7k759\""
Sep 17 15:53:57.647705 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.647690    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:53:57.648144 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.648126    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.648894 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.648875    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:53:57.648990 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.648878    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:53:57.648990 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.648934    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:53:57.649096 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.648878    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lvsrt\""
Sep 17 15:53:57.650568 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.650544    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:53:57.651243 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.651225    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wn8rd\""
Sep 17 15:53:57.651777 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.651756    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:53:57.651876 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.651861    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:53:57.651963 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.651862    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:53:57.652788 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.652764    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:53:57.653569 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.653017    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bq7m8\""
Sep 17 15:53:57.653569 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.653140    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.653569 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.653150    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:53:57.656165 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.655745    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.656645 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.656628    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:53:57.656747 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.656669    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:53:57.656806 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.656749    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f22p2\""
Sep 17 15:53:57.658726 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.658708    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.659388 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.659190    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:53:57.659388 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.659262    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:53:57.659505 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.659437    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g62kp\""
Sep 17 15:53:57.660747 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.660730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:53:57.660824 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.660806    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:53:57.661530 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.661513    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:53:57.663251 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.663222    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mdp2n\""
Sep 17 15:53:57.663355 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.663271    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:53:57.663419 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.663409    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:53:57.672396 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.672353    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:53:57.674488 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.673737    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:53:57.683750 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683728    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-sys\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.683835 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683764    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f843638-5eea-4ff9-99e4-33229d7815a3-kubelet-config\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.683835 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683793    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-cni-binary-copy\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.683835 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683817    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-hostroot\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.683955 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683837    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-conf-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.683955 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-lib-modules\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.683955 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683868    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-host\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.683955 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683909    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-cnibin\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.683955 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683946    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-slash\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683967    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-ovnkube-config\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683980    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-device-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.684098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.683995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-daemon-config\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.684098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684022    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fccl\" (UniqueName: \"kubernetes.io/projected/74438d26-e6ab-4d96-a541-0f428f8951f2-kube-api-access-4fccl\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.684098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684045    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.684098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684061    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-env-overrides\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-kubernetes\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684165    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-systemd\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684189    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-tmp\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684212    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684236    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-system-cni-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684261    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-multus-certs\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684303    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-systemd\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684326    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-node-log\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684348    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684369    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f843638-5eea-4ff9-99e4-33229d7815a3-dbus\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.684411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684399    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684438    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-socket-dir-parent\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684487    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4gtn\" (UniqueName: \"kubernetes.io/projected/bb38688f-9e3c-4245-b721-c31842c74f70-kube-api-access-k4gtn\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-systemd-units\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684522    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-run-netns\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684536    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7cc\" (UniqueName: \"kubernetes.io/projected/d601c0c7-3f72-4f8d-876f-cf772720348b-kube-api-access-kq7cc\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684557    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-etc-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684594    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-cni-netd\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684633    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlp72\" (UniqueName: \"kubernetes.io/projected/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-kube-api-access-vlp72\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684663    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/572c0334-33e6-444a-80e3-caa407bd897d-agent-certs\") pod \"konnectivity-agent-s6qlb\" (UID: \"572c0334-33e6-444a-80e3-caa407bd897d\") " pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684692    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysconfig\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684714    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr6b\" (UniqueName: \"kubernetes.io/projected/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-kube-api-access-dtr6b\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684739    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-ovn\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684763    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684788    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d601c0c7-3f72-4f8d-876f-cf772720348b-iptables-alerter-script\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684810    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.684868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684832    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-tuned\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684872    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-log-socket\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684919    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/572c0334-33e6-444a-80e3-caa407bd897d-konnectivity-ca\") pod \"konnectivity-agent-s6qlb\" (UID: \"572c0334-33e6-444a-80e3-caa407bd897d\") " pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684941    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684954    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-run\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684967    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-cni-binary-copy\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.684991    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685013    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-cni-bin\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685033    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-kubelet\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685047    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74438d26-e6ab-4d96-a541-0f428f8951f2-host\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685112    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxc7\" (UniqueName: \"kubernetes.io/projected/afc9f293-dabb-4bff-b819-41dc8625ce82-kube-api-access-nmxc7\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685137    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysctl-d\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685160    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysctl-conf\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685205    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-var-lib-kubelet\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.685387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685225    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-etc-kubernetes\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685263    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-cnibin\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685321    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-cni-multus\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685337    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-ovnkube-script-lib\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685351    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-sys-fs\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685365    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/74438d26-e6ab-4d96-a541-0f428f8951f2-serviceca\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685379    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jtk\" (UniqueName: \"kubernetes.io/projected/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-kube-api-access-k4jtk\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685396    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-k8s-cni-cncf-io\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685411    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-cni-bin\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685445    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb38688f-9e3c-4245-b721-c31842c74f70-ovn-node-metrics-cert\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685473    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-hosts-file\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685494    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-tmp-dir\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685520    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq62m\" (UniqueName: \"kubernetes.io/projected/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-kube-api-access-pq62m\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685541    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-socket-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685564    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-registration-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685585    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-modprobe-d\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.685847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685599    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-system-cni-dir\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.686279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfll\" (UniqueName: \"kubernetes.io/projected/d6c09c6b-33dd-4de9-aab8-222a50919654-kube-api-access-mdfll\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.686279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685664    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-cni-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.686279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685689    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-netns\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.686279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685710    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-kubelet\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.686279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685728    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-var-lib-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.686279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685744    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.686279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.685764    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d601c0c7-3f72-4f8d-876f-cf772720348b-host-slash\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.686598 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.686498    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:53:57.734746 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.734724    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rvvj2"
Sep 17 15:53:57.756540 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.756517    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rvvj2"
Sep 17 15:53:57.781160 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.781080    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" event={"ID":"acce870b27226e32c83f4f167afc6e1c","Type":"ContainerStarted","Data":"d850e34987bbaa22a69d395e8a8668ca8b0f902f2bde158c7beb50b8a7e26623"}
Sep 17 15:53:57.782505 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.782470    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" event={"ID":"3bca85661d4e2826a71d7270087fe9ca","Type":"ContainerStarted","Data":"dd6918da1be15d7f02947f7107a1f9552545d602db8a3ed4b6acfb9791d97e21"}
Sep 17 15:53:57.786323 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.786419 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-system-cni-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.786419 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786360    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-multus-certs\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.786419 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786382    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-systemd\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786552 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786439    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-systemd\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786552 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.786447    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:53:57.786552 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786439    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-multus-certs\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.786552 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786472    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-node-log\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786552 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786474    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-system-cni-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.786552 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786508    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-node-log\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786552 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.786530    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret podName:0f843638-5eea-4ff9-99e4-33229d7815a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:53:58.286497212 +0000 UTC m=+4.552012211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret") pod "global-pull-secret-syncer-q4f6r" (UID: "0f843638-5eea-4ff9-99e4-33229d7815a3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786565    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786588    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f843638-5eea-4ff9-99e4-33229d7815a3-dbus\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786614    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.786627    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786641    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-socket-dir-parent\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.786668    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs podName:afc9f293-dabb-4bff-b819-41dc8625ce82 nodeName:}" failed. No retries permitted until 2026-09-17 15:53:58.286653287 +0000 UTC m=+4.552168278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs") pod "network-metrics-daemon-xh8xc" (UID: "afc9f293-dabb-4bff-b819-41dc8625ce82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786686    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gtn\" (UniqueName: \"kubernetes.io/projected/bb38688f-9e3c-4245-b721-c31842c74f70-kube-api-access-k4gtn\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786710    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-systemd-units\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786729    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-run-netns\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786754    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7cc\" (UniqueName: \"kubernetes.io/projected/d601c0c7-3f72-4f8d-876f-cf772720348b-kube-api-access-kq7cc\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786777    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-etc-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786782    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-socket-dir-parent\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786799    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f843638-5eea-4ff9-99e4-33229d7815a3-dbus\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786804    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-cni-netd\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786837    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlp72\" (UniqueName: \"kubernetes.io/projected/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-kube-api-access-vlp72\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786867    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-cni-netd\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.786936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786871    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/572c0334-33e6-444a-80e3-caa407bd897d-agent-certs\") pod \"konnectivity-agent-s6qlb\" (UID: \"572c0334-33e6-444a-80e3-caa407bd897d\") " pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786908    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-systemd-units\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786912    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysconfig\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786950    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr6b\" (UniqueName: \"kubernetes.io/projected/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-kube-api-access-dtr6b\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786961    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-etc-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.786977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-ovn\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787000    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-run-netns\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787018    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787045    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d601c0c7-3f72-4f8d-876f-cf772720348b-iptables-alerter-script\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787050    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysconfig\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787069    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787094    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-tuned\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787115    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-log-socket\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/572c0334-33e6-444a-80e3-caa407bd897d-konnectivity-ca\") pod \"konnectivity-agent-s6qlb\" (UID: \"572c0334-33e6-444a-80e3-caa407bd897d\") " pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787144    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787188    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-run\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.787732 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-cni-binary-copy\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787243    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-cni-bin\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787317    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-kubelet\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787342    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787364    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787392    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787414    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74438d26-e6ab-4d96-a541-0f428f8951f2-host\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787436    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxc7\" (UniqueName: \"kubernetes.io/projected/afc9f293-dabb-4bff-b819-41dc8625ce82-kube-api-access-nmxc7\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787455    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysctl-d\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787474    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysctl-conf\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787446    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787502    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-var-lib-kubelet\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787524    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-etc-kubernetes\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787553    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-cnibin\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787575    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-cni-multus\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787593    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-kubelet\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.788478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787689    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-var-lib-kubelet\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787740    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787742    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-etc-kubernetes\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787779    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/74438d26-e6ab-4d96-a541-0f428f8951f2-host\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787797    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-cnibin\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787842    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-cni-multus\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787953    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysctl-conf\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788031    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-sysctl-d\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788085    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d601c0c7-3f72-4f8d-876f-cf772720348b-iptables-alerter-script\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788170    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-var-lib-cni-bin\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788230    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-run\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788242    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-ovnkube-script-lib\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787214    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-ovn\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.787598    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-ovnkube-script-lib\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788312    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-log-socket\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788336    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-sys-fs\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788356    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-run-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.789442 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788363    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/74438d26-e6ab-4d96-a541-0f428f8951f2-serviceca\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788400    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jtk\" (UniqueName: \"kubernetes.io/projected/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-kube-api-access-k4jtk\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788419    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-sys-fs\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788426    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-k8s-cni-cncf-io\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788451    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-cni-bin\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788474    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb38688f-9e3c-4245-b721-c31842c74f70-ovn-node-metrics-cert\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788497    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-hosts-file\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788520    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-tmp-dir\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788529    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-cni-binary-copy\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788543    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq62m\" (UniqueName: \"kubernetes.io/projected/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-kube-api-access-pq62m\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-socket-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788599    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/572c0334-33e6-444a-80e3-caa407bd897d-konnectivity-ca\") pod \"konnectivity-agent-s6qlb\" (UID: \"572c0334-33e6-444a-80e3-caa407bd897d\") " pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788602    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-registration-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-k8s-cni-cncf-io\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788654    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-registration-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-modprobe-d\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-hosts-file\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.790362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788687    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-system-cni-dir\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788794    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-modprobe-d\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788840    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-system-cni-dir\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788869    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/74438d26-e6ab-4d96-a541-0f428f8951f2-serviceca\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.788914    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-cni-bin\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789001    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-tmp-dir\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789030    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-socket-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789047    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfll\" (UniqueName: \"kubernetes.io/projected/d6c09c6b-33dd-4de9-aab8-222a50919654-kube-api-access-mdfll\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789069    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-cni-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789083    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-netns\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789097    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-kubelet\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789120    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-var-lib-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789147    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789157    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-cni-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d601c0c7-3f72-4f8d-876f-cf772720348b-host-slash\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789197    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789232    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-kubelet\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789251    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-sys\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.791161 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789274    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d601c0c7-3f72-4f8d-876f-cf772720348b-host-slash\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789242    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-var-lib-openvswitch\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789303    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f843638-5eea-4ff9-99e4-33229d7815a3-kubelet-config\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789317    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-host-run-netns\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-cni-binary-copy\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789308    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-sys\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789347    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f843638-5eea-4ff9-99e4-33229d7815a3-kubelet-config\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789364    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-hostroot\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c09c6b-33dd-4de9-aab8-222a50919654-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789392    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-conf-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789420    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-hostroot\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789428    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-lib-modules\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789432    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-conf-dir\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789454    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-host\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789480    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-cnibin\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789504    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-slash\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789521    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-lib-modules\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789525    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-ovnkube-config\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.791897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789541    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-host\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789562    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-device-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-daemon-config\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fccl\" (UniqueName: \"kubernetes.io/projected/74438d26-e6ab-4d96-a541-0f428f8951f2-kube-api-access-4fccl\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-device-dir\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789646    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789672    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-env-overrides\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789699    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-kubernetes\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-systemd\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789747    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-tmp\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789814    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-cni-binary-copy\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789820    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb38688f-9e3c-4245-b721-c31842c74f70-host-slash\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789858    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-ovnkube-config\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789909    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-kubernetes\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789975    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c09c6b-33dd-4de9-aab8-222a50919654-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.790047    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-systemd\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.789595    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-cnibin\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.790240    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb38688f-9e3c-4245-b721-c31842c74f70-env-overrides\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.792491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.790461    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-multus-daemon-config\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.793084 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.791876    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-etc-tuned\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.793084 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.791891    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-tmp\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.793084 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.791950    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/572c0334-33e6-444a-80e3-caa407bd897d-agent-certs\") pod \"konnectivity-agent-s6qlb\" (UID: \"572c0334-33e6-444a-80e3-caa407bd897d\") " pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:53:57.793084 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.792075    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb38688f-9e3c-4245-b721-c31842c74f70-ovn-node-metrics-cert\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.811104 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.811038    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fccl\" (UniqueName: \"kubernetes.io/projected/74438d26-e6ab-4d96-a541-0f428f8951f2-kube-api-access-4fccl\") pod \"node-ca-prh25\" (UID: \"74438d26-e6ab-4d96-a541-0f428f8951f2\") " pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:57.813253 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.812724    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:53:57.813253 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.812750    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:53:57.813253 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.812765    2585 projected.go:196] Error preparing data for projected volume kube-api-access-kj4bl for pod openshift-network-diagnostics/network-check-target-29lpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:53:57.813253 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:57.812934    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl podName:d398cd52-c7fe-4df9-832c-1c0a2c357e45 nodeName:}" failed. No retries permitted until 2026-09-17 15:53:58.31291179 +0000 UTC m=+4.578426791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kj4bl" (UniqueName: "kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl") pod "network-check-target-29lpf" (UID: "d398cd52-c7fe-4df9-832c-1c0a2c357e45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:53:57.814486 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.814467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq62m\" (UniqueName: \"kubernetes.io/projected/34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c-kube-api-access-pq62m\") pod \"multus-x5njn\" (UID: \"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c\") " pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.814729 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.814708    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7cc\" (UniqueName: \"kubernetes.io/projected/d601c0c7-3f72-4f8d-876f-cf772720348b-kube-api-access-kq7cc\") pod \"iptables-alerter-ht2p6\" (UID: \"d601c0c7-3f72-4f8d-876f-cf772720348b\") " pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.815414 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.815380    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxc7\" (UniqueName: \"kubernetes.io/projected/afc9f293-dabb-4bff-b819-41dc8625ce82-kube-api-access-nmxc7\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:57.817120 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.817100    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlp72\" (UniqueName: \"kubernetes.io/projected/b407d987-6f5a-46c9-b0cf-ef949bbf19ce-kube-api-access-vlp72\") pod \"aws-ebs-csi-driver-node-rghxj\" (UID: \"b407d987-6f5a-46c9-b0cf-ef949bbf19ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.817383 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.817367    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jtk\" (UniqueName: \"kubernetes.io/projected/64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb-kube-api-access-k4jtk\") pod \"node-resolver-ksfvw\" (UID: \"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb\") " pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:57.825245 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.825226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr6b\" (UniqueName: \"kubernetes.io/projected/855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4-kube-api-access-dtr6b\") pod \"tuned-nxskj\" (UID: \"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4\") " pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.825642 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.825622    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gtn\" (UniqueName: \"kubernetes.io/projected/bb38688f-9e3c-4245-b721-c31842c74f70-kube-api-access-k4gtn\") pod \"ovnkube-node-9ttzj\" (UID: \"bb38688f-9e3c-4245-b721-c31842c74f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.825922 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.825902    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfll\" (UniqueName: \"kubernetes.io/projected/d6c09c6b-33dd-4de9-aab8-222a50919654-kube-api-access-mdfll\") pod \"multus-additional-cni-plugins-s28wz\" (UID: \"d6c09c6b-33dd-4de9-aab8-222a50919654\") " pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.947834 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.947749    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ht2p6"
Sep 17 15:53:57.954851 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.954831    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x5njn"
Sep 17 15:53:57.963541 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.963523    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nxskj"
Sep 17 15:53:57.969370 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.969328    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:53:57.975672 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.975653    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj"
Sep 17 15:53:57.983153 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.983122    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:53:57.990698 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.990451    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s28wz"
Sep 17 15:53:57.997063 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:57.996930    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-prh25"
Sep 17 15:53:58.001751 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:58.001617    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ksfvw"
Sep 17 15:53:58.061542 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:58.061520    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:53:58.294146 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:58.294110    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:58.294319 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:58.294162    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:58.294319 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:58.294311    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:53:58.294419 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:58.294377    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs podName:afc9f293-dabb-4bff-b819-41dc8625ce82 nodeName:}" failed. No retries permitted until 2026-09-17 15:53:59.294356696 +0000 UTC m=+5.559871694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs") pod "network-metrics-daemon-xh8xc" (UID: "afc9f293-dabb-4bff-b819-41dc8625ce82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:53:58.294477 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:58.294447    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:53:58.294515 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:58.294481    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret podName:0f843638-5eea-4ff9-99e4-33229d7815a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:53:59.294469766 +0000 UTC m=+5.559984761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret") pod "global-pull-secret-syncer-q4f6r" (UID: "0f843638-5eea-4ff9-99e4-33229d7815a3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:53:58.394539 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:58.394502    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:53:58.394723 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:58.394635    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:53:58.394723 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:58.394657    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:53:58.394723 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:58.394669    2585 projected.go:196] Error preparing data for projected volume kube-api-access-kj4bl for pod openshift-network-diagnostics/network-check-target-29lpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:53:58.394723 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:58.394725    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl podName:d398cd52-c7fe-4df9-832c-1c0a2c357e45 nodeName:}" failed. No retries permitted until 2026-09-17 15:53:59.394707965 +0000 UTC m=+5.660222966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj4bl" (UniqueName: "kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl") pod "network-check-target-29lpf" (UID: "d398cd52-c7fe-4df9-832c-1c0a2c357e45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:53:58.760186 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:58.760123    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:48:57 +0000 UTC" deadline="2028-04-18 01:56:14.278131604 +0000 UTC"
Sep 17 15:53:58.760186 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:58.760188    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13882h2m15.517952458s"
Sep 17 15:53:59.061857 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.061829    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64748e47_f5f7_47d7_a2cb_0a5ff8ffd3fb.slice/crio-b08c8460679796f23c91707f6a4b24cb2bd679fa449ae167ffef18d0ba119731 WatchSource:0}: Error finding container b08c8460679796f23c91707f6a4b24cb2bd679fa449ae167ffef18d0ba119731: Status 404 returned error can't find the container with id b08c8460679796f23c91707f6a4b24cb2bd679fa449ae167ffef18d0ba119731
Sep 17 15:53:59.063234 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.063210    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34afb8a0_7a2b_4aa8_9eae_d11acf59fe8c.slice/crio-e0b10a9e85b6442edb2e1edb7663fc85ce8a8eadc1ef18f33025dc1caf846a45 WatchSource:0}: Error finding container e0b10a9e85b6442edb2e1edb7663fc85ce8a8eadc1ef18f33025dc1caf846a45: Status 404 returned error can't find the container with id e0b10a9e85b6442edb2e1edb7663fc85ce8a8eadc1ef18f33025dc1caf846a45
Sep 17 15:53:59.064903 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.064880    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb407d987_6f5a_46c9_b0cf_ef949bbf19ce.slice/crio-10f93969e32dbd5c050239ac321f5c567206f6e711bec960e6caec23e27ffaf4 WatchSource:0}: Error finding container 10f93969e32dbd5c050239ac321f5c567206f6e711bec960e6caec23e27ffaf4: Status 404 returned error can't find the container with id 10f93969e32dbd5c050239ac321f5c567206f6e711bec960e6caec23e27ffaf4
Sep 17 15:53:59.086493 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.086467    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74438d26_e6ab_4d96_a541_0f428f8951f2.slice/crio-24bd05f170f27c6b427f14055808096e777f9651738c85efbdc5fb257fd2f770 WatchSource:0}: Error finding container 24bd05f170f27c6b427f14055808096e777f9651738c85efbdc5fb257fd2f770: Status 404 returned error can't find the container with id 24bd05f170f27c6b427f14055808096e777f9651738c85efbdc5fb257fd2f770
Sep 17 15:53:59.087133 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.087048    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c09c6b_33dd_4de9_aab8_222a50919654.slice/crio-d8761c7b10ec5c72117438f63655be3747547e082a93048dc58c38771a2e308e WatchSource:0}: Error finding container d8761c7b10ec5c72117438f63655be3747547e082a93048dc58c38771a2e308e: Status 404 returned error can't find the container with id d8761c7b10ec5c72117438f63655be3747547e082a93048dc58c38771a2e308e
Sep 17 15:53:59.088066 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.087968    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572c0334_33e6_444a_80e3_caa407bd897d.slice/crio-1e7fe9d66dc5ade233dca7ef39e0f8d7f6ea61d5422a20570db5621e0d7e1d59 WatchSource:0}: Error finding container 1e7fe9d66dc5ade233dca7ef39e0f8d7f6ea61d5422a20570db5621e0d7e1d59: Status 404 returned error can't find the container with id 1e7fe9d66dc5ade233dca7ef39e0f8d7f6ea61d5422a20570db5621e0d7e1d59
Sep 17 15:53:59.091164 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.088882    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855b8cb7_2cd7_46cb_be5c_89b33bfcd9a4.slice/crio-69fab95cd7ffb46cfbf550ee804ec0e9d26d8f2dfba793f8fc2c124bc84a14a6 WatchSource:0}: Error finding container 69fab95cd7ffb46cfbf550ee804ec0e9d26d8f2dfba793f8fc2c124bc84a14a6: Status 404 returned error can't find the container with id 69fab95cd7ffb46cfbf550ee804ec0e9d26d8f2dfba793f8fc2c124bc84a14a6
Sep 17 15:53:59.091164 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.089744    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb38688f_9e3c_4245_b721_c31842c74f70.slice/crio-4dc88ce68eb7281b08ed5a8456dc71b9ee0012e3e55436532e234079d5ca0e7a WatchSource:0}: Error finding container 4dc88ce68eb7281b08ed5a8456dc71b9ee0012e3e55436532e234079d5ca0e7a: Status 404 returned error can't find the container with id 4dc88ce68eb7281b08ed5a8456dc71b9ee0012e3e55436532e234079d5ca0e7a
Sep 17 15:53:59.091164 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:53:59.090594    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd601c0c7_3f72_4f8d_876f_cf772720348b.slice/crio-fa781afcd091b622ebbe9d28baf721ce9d0a44fd776bce71718736e8fd203297 WatchSource:0}: Error finding container fa781afcd091b622ebbe9d28baf721ce9d0a44fd776bce71718736e8fd203297: Status 404 returned error can't find the container with id fa781afcd091b622ebbe9d28baf721ce9d0a44fd776bce71718736e8fd203297
Sep 17 15:53:59.300261 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.300235    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:59.300411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.300274    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:59.300411 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.300401    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:53:59.300535 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.300412    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:53:59.300535 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.300471    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs podName:afc9f293-dabb-4bff-b819-41dc8625ce82 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:01.30044874 +0000 UTC m=+7.565963730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs") pod "network-metrics-daemon-xh8xc" (UID: "afc9f293-dabb-4bff-b819-41dc8625ce82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:53:59.300535 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.300491    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret podName:0f843638-5eea-4ff9-99e4-33229d7815a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:01.300479562 +0000 UTC m=+7.565994539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret") pod "global-pull-secret-syncer-q4f6r" (UID: "0f843638-5eea-4ff9-99e4-33229d7815a3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:53:59.400801 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.400601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:53:59.400900 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.400766    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:53:59.400900 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.400865    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:53:59.400900 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.400878    2585 projected.go:196] Error preparing data for projected volume kube-api-access-kj4bl for pod openshift-network-diagnostics/network-check-target-29lpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:53:59.400999 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.400934    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl podName:d398cd52-c7fe-4df9-832c-1c0a2c357e45 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:01.400918211 +0000 UTC m=+7.666433209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj4bl" (UniqueName: "kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl") pod "network-check-target-29lpf" (UID: "d398cd52-c7fe-4df9-832c-1c0a2c357e45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:53:59.760454 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.760371    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:48:57 +0000 UTC" deadline="2028-04-03 03:04:02.871595011 +0000 UTC"
Sep 17 15:53:59.760454 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.760445    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13523h10m3.111153593s"
Sep 17 15:53:59.779014 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.778986    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:53:59.779155 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.779110    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:53:59.779518 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.779499    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:53:59.779629 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.779610    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:53:59.779705 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.779672    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:53:59.779767 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:53:59.779739    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:53:59.800731 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.800699    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" event={"ID":"3bca85661d4e2826a71d7270087fe9ca","Type":"ContainerStarted","Data":"97055ad16e3940978bde5b3c25c47238fa9613d9e6842714cbb9a75435fb56d2"}
Sep 17 15:53:59.814156 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.814129    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"4dc88ce68eb7281b08ed5a8456dc71b9ee0012e3e55436532e234079d5ca0e7a"}
Sep 17 15:53:59.827783 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.827759    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nxskj" event={"ID":"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4","Type":"ContainerStarted","Data":"69fab95cd7ffb46cfbf550ee804ec0e9d26d8f2dfba793f8fc2c124bc84a14a6"}
Sep 17 15:53:59.837880 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.837858    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj" event={"ID":"b407d987-6f5a-46c9-b0cf-ef949bbf19ce","Type":"ContainerStarted","Data":"10f93969e32dbd5c050239ac321f5c567206f6e711bec960e6caec23e27ffaf4"}
Sep 17 15:53:59.846010 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.845984    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ksfvw" event={"ID":"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb","Type":"ContainerStarted","Data":"b08c8460679796f23c91707f6a4b24cb2bd679fa449ae167ffef18d0ba119731"}
Sep 17 15:53:59.852351 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.852264    2585 generic.go:356] "Generic (PLEG): container finished" podID="acce870b27226e32c83f4f167afc6e1c" containerID="8128bafc5c0edadb92b98d4c64ad59a7919d921d1bf205a62596aad18af96366" exitCode=0
Sep 17 15:53:59.852706 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.852333    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" event={"ID":"acce870b27226e32c83f4f167afc6e1c","Type":"ContainerDied","Data":"8128bafc5c0edadb92b98d4c64ad59a7919d921d1bf205a62596aad18af96366"}
Sep 17 15:53:59.865639 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.865570    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ht2p6" event={"ID":"d601c0c7-3f72-4f8d-876f-cf772720348b","Type":"ContainerStarted","Data":"fa781afcd091b622ebbe9d28baf721ce9d0a44fd776bce71718736e8fd203297"}
Sep 17 15:53:59.874298 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.874261    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s6qlb" event={"ID":"572c0334-33e6-444a-80e3-caa407bd897d","Type":"ContainerStarted","Data":"1e7fe9d66dc5ade233dca7ef39e0f8d7f6ea61d5422a20570db5621e0d7e1d59"}
Sep 17 15:53:59.877155 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.877107    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-18.ec2.internal" podStartSLOduration=3.877089584 podStartE2EDuration="3.877089584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:59.823078759 +0000 UTC m=+6.088593759" watchObservedRunningTime="2026-09-17 15:53:59.877089584 +0000 UTC m=+6.142604586"
Sep 17 15:53:59.884551 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.884526    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s28wz" event={"ID":"d6c09c6b-33dd-4de9-aab8-222a50919654","Type":"ContainerStarted","Data":"d8761c7b10ec5c72117438f63655be3747547e082a93048dc58c38771a2e308e"}
Sep 17 15:53:59.898232 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.898205    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-prh25" event={"ID":"74438d26-e6ab-4d96-a541-0f428f8951f2","Type":"ContainerStarted","Data":"24bd05f170f27c6b427f14055808096e777f9651738c85efbdc5fb257fd2f770"}
Sep 17 15:53:59.907742 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:53:59.907700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x5njn" event={"ID":"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c","Type":"ContainerStarted","Data":"e0b10a9e85b6442edb2e1edb7663fc85ce8a8eadc1ef18f33025dc1caf846a45"}
Sep 17 15:54:00.933516 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:00.933478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" event={"ID":"acce870b27226e32c83f4f167afc6e1c","Type":"ContainerStarted","Data":"590e656d90c97dbbea2ef1307338b4416cd9794c2affa5c38601e816fab3f87d"}
Sep 17 15:54:01.316822 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:01.316023    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:01.316822 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:01.316075    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:01.316822 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.316251    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:01.316822 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.316329    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs podName:afc9f293-dabb-4bff-b819-41dc8625ce82 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:05.316310294 +0000 UTC m=+11.581825292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs") pod "network-metrics-daemon-xh8xc" (UID: "afc9f293-dabb-4bff-b819-41dc8625ce82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:01.316822 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.316734    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:01.316822 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.316789    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret podName:0f843638-5eea-4ff9-99e4-33229d7815a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:05.316769819 +0000 UTC m=+11.582284800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret") pod "global-pull-secret-syncer-q4f6r" (UID: "0f843638-5eea-4ff9-99e4-33229d7815a3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:01.417321 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:01.417268    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:01.417536 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.417500    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:01.417536 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.417524    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:01.417536 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.417537    2585 projected.go:196] Error preparing data for projected volume kube-api-access-kj4bl for pod openshift-network-diagnostics/network-check-target-29lpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:01.417762 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.417593    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl podName:d398cd52-c7fe-4df9-832c-1c0a2c357e45 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:05.417574698 +0000 UTC m=+11.683089690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj4bl" (UniqueName: "kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl") pod "network-check-target-29lpf" (UID: "d398cd52-c7fe-4df9-832c-1c0a2c357e45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:01.784517 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:01.783889    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:01.784517 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:01.783917    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:01.784517 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:01.783921    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:01.784517 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.784041    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:01.784517 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.784188    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:01.784517 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:01.784305    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:03.778601 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:03.777607    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:03.778601 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:03.777600    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:03.778601 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:03.777904    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:03.778601 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:03.777989    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:03.778601 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:03.778113    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:03.778601 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:03.778211    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:05.345400 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:05.345357    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:05.345940 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:05.345421    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:05.345940 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.345537    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:05.345940 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.345574    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:05.345940 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.345612    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret podName:0f843638-5eea-4ff9-99e4-33229d7815a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:13.345595421 +0000 UTC m=+19.611110415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret") pod "global-pull-secret-syncer-q4f6r" (UID: "0f843638-5eea-4ff9-99e4-33229d7815a3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:05.345940 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.345678    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs podName:afc9f293-dabb-4bff-b819-41dc8625ce82 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:13.345633689 +0000 UTC m=+19.611148681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs") pod "network-metrics-daemon-xh8xc" (UID: "afc9f293-dabb-4bff-b819-41dc8625ce82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:05.446402 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:05.446364    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:05.446581 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.446503    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:05.446581 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.446525    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:05.446581 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.446536    2585 projected.go:196] Error preparing data for projected volume kube-api-access-kj4bl for pod openshift-network-diagnostics/network-check-target-29lpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:05.446713 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.446585    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl podName:d398cd52-c7fe-4df9-832c-1c0a2c357e45 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:13.446572309 +0000 UTC m=+19.712087301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj4bl" (UniqueName: "kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl") pod "network-check-target-29lpf" (UID: "d398cd52-c7fe-4df9-832c-1c0a2c357e45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:05.778620 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:05.778586    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:05.778916 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.778704    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:05.779072 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:05.779055    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:05.779166 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.779148    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:05.779314 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:05.779273    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:05.779438 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:05.779375    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:07.778351 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:07.777650    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:07.778351 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:07.777788    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:07.778351 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:07.778202    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:07.778351 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:07.778305    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:07.779258 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:07.779120    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:07.779258 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:07.779224    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:09.777938 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:09.776826    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:09.777938 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:09.776938    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:09.777938 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:09.777529    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:09.777938 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:09.777601    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:09.777938 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:09.777653    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:09.777938 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:09.777698    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:11.777709 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:11.777665    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:11.778151 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:11.777665    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:11.778151 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:11.777803    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:11.778151 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:11.777665    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:11.778151 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:11.777893    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:11.778151 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:11.777978    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:13.405434 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:13.405379    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:13.405434 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:13.405439    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:13.405871 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.405548    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:13.405871 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.405572    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:13.405871 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.405627    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret podName:0f843638-5eea-4ff9-99e4-33229d7815a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:29.405606275 +0000 UTC m=+35.671121264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret") pod "global-pull-secret-syncer-q4f6r" (UID: "0f843638-5eea-4ff9-99e4-33229d7815a3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:13.405871 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.405651    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs podName:afc9f293-dabb-4bff-b819-41dc8625ce82 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:29.405639905 +0000 UTC m=+35.671154898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs") pod "network-metrics-daemon-xh8xc" (UID: "afc9f293-dabb-4bff-b819-41dc8625ce82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:13.506526 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:13.506482    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:13.506697 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.506651    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:13.506697 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.506670    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:13.506697 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.506679    2585 projected.go:196] Error preparing data for projected volume kube-api-access-kj4bl for pod openshift-network-diagnostics/network-check-target-29lpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:13.506836 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.506737    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl podName:d398cd52-c7fe-4df9-832c-1c0a2c357e45 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:29.506716275 +0000 UTC m=+35.772231261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj4bl" (UniqueName: "kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl") pod "network-check-target-29lpf" (UID: "d398cd52-c7fe-4df9-832c-1c0a2c357e45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:13.780620 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:13.780594    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:13.780814 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:13.780594    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:13.780814 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.780723    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:13.780814 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:13.780594    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:13.780814 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.780792    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:13.781004 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:13.780902    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:15.780662 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:15.780633    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:15.781101 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:15.780633    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:15.781101 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:15.780757    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:15.781101 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:15.780772    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:15.781101 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:15.780877    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:15.781101 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:15.780938    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:16.985880 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:16.985138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s6qlb" event={"ID":"572c0334-33e6-444a-80e3-caa407bd897d","Type":"ContainerStarted","Data":"db41d024de17836f90f363381caeee5429bb456b780dbaffa70fbe25162bf9bf"}
Sep 17 15:54:17.006183 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.005739    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s6qlb" podStartSLOduration=7.430552002 podStartE2EDuration="21.005271128s" totalImagesPullingTime="13.574719126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:17.004528194 +0000 UTC m=+23.270043195" watchObservedRunningTime="2026-09-17 15:54:17.005271128 +0000 UTC m=+23.270786129"
Sep 17 15:54:17.006183 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.005983    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-18.ec2.internal" podStartSLOduration=20.005969653 podStartE2EDuration="20.005969653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:00.96211169 +0000 UTC m=+7.227626693" watchObservedRunningTime="2026-09-17 15:54:17.005969653 +0000 UTC m=+23.271484655"
Sep 17 15:54:17.777983 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.777786    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:17.778306 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.777787    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:17.778306 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:17.778084    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:17.778306 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.777787    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:17.778306 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:17.778150    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:17.778306 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:17.778216    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:17.990313 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.990123    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"fa3991b4039c79cc58de18fb888eb9b0dbb8bef56b1e879ab6d5bf859eea241b"}
Sep 17 15:54:17.990951 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.990380    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"6c487e9c06a40bd40c1955f590d874bfdfa9d283dad24785bfa1560d78e15e56"}
Sep 17 15:54:17.990951 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.990420    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"c9474fcf374af4149d26df7785810074b003c6b8cc16de43983c01c5984bf229"}
Sep 17 15:54:17.990951 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.990457    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"fcd6b87bca8ac74cfcb526d5da09d3a5b621709b81f9b2749c40b40093f46574"}
Sep 17 15:54:17.990951 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.990492    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"914719e759f28d2994090b0769b6279777dd6ad542ff7eab5164a4010e05be5f"}
Sep 17 15:54:17.990951 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.990524    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"7470b65a4a0b0ef1f5921b9d84745a55d45ca2a3d73e5acc485d84cff525c0ba"}
Sep 17 15:54:17.993438 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.992128    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nxskj" event={"ID":"855b8cb7-2cd7-46cb-be5c-89b33bfcd9a4","Type":"ContainerStarted","Data":"c4f3a038933b2860a1ecefdde27de5fc9db424953e54f39139212e3a2df69262"}
Sep 17 15:54:17.995256 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.995232    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj" event={"ID":"b407d987-6f5a-46c9-b0cf-ef949bbf19ce","Type":"ContainerStarted","Data":"10267153acbb41ee846e7e181e06295aad67fe57d6fcf2d20aa28114d7ad620b"}
Sep 17 15:54:17.996850 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.996823    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ksfvw" event={"ID":"64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb","Type":"ContainerStarted","Data":"d517f789f3496546e1b7db4addef890c33867145b0cf9b01c012c627d295f6a9"}
Sep 17 15:54:17.998245 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.998225    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6c09c6b-33dd-4de9-aab8-222a50919654" containerID="cbbca99f8dfbc33d1ee756f306e1bae6e1459d6936d06020b45cad1f51e22207" exitCode=0
Sep 17 15:54:17.998351 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.998304    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s28wz" event={"ID":"d6c09c6b-33dd-4de9-aab8-222a50919654","Type":"ContainerDied","Data":"cbbca99f8dfbc33d1ee756f306e1bae6e1459d6936d06020b45cad1f51e22207"}
Sep 17 15:54:17.999885 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:17.999868    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-prh25" event={"ID":"74438d26-e6ab-4d96-a541-0f428f8951f2","Type":"ContainerStarted","Data":"5873dad7404dc27865a5215c2f2523fc9e92bedeae6856267d076c20e577e309"}
Sep 17 15:54:18.001719 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.001700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x5njn" event={"ID":"34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c","Type":"ContainerStarted","Data":"f12db4b98fadfd678dc130c7f999a69d59e21fa16bb7caec08e5f6980814e2ae"}
Sep 17 15:54:18.015318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.015297    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:54:18.016263 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.016220    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nxskj" podStartSLOduration=4.290557496 podStartE2EDuration="22.016207263s" totalImagesPullingTime="17.725649767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:18.016102389 +0000 UTC m=+24.281617388" watchObservedRunningTime="2026-09-17 15:54:18.016207263 +0000 UTC m=+24.281722262"
Sep 17 15:54:18.067306 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.067246    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x5njn" podStartSLOduration=5.293347623 podStartE2EDuration="23.067234238s" totalImagesPullingTime="17.773886615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:18.040778043 +0000 UTC m=+24.306293044" watchObservedRunningTime="2026-09-17 15:54:18.067234238 +0000 UTC m=+24.332749237"
Sep 17 15:54:18.148117 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.148084    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-prh25" podStartSLOduration=4.678466154 podStartE2EDuration="22.148071407s" totalImagesPullingTime="17.469605253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:18.099849289 +0000 UTC m=+24.365364288" watchObservedRunningTime="2026-09-17 15:54:18.148071407 +0000 UTC m=+24.413586402"
Sep 17 15:54:18.148631 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.148609    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ksfvw" podStartSLOduration=4.440560339 podStartE2EDuration="22.148603578s" totalImagesPullingTime="17.708043239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:18.147961567 +0000 UTC m=+24.413476567" watchObservedRunningTime="2026-09-17 15:54:18.148603578 +0000 UTC m=+24.414118577"
Sep 17 15:54:18.748867 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.748776    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:54:18.015314141Z","UUID":"8fde20a2-0503-433e-9d17-12951dc8d023","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:54:18.750401 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.750379    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:54:18.750527 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:18.750424    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:54:19.005590 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:19.005507    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ht2p6" event={"ID":"d601c0c7-3f72-4f8d-876f-cf772720348b","Type":"ContainerStarted","Data":"0f040d969ef5f564623fb2dee0a6e42de2d39ebe71ad990ce5c9253ef70355e6"}
Sep 17 15:54:19.008583 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:19.008553    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj" event={"ID":"b407d987-6f5a-46c9-b0cf-ef949bbf19ce","Type":"ContainerStarted","Data":"bfc02409c6774cc348d52b5c5d452cb2d75b7e2720a6927de8fca1fb13782908"}
Sep 17 15:54:19.008701 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:19.008608    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj" event={"ID":"b407d987-6f5a-46c9-b0cf-ef949bbf19ce","Type":"ContainerStarted","Data":"9a05bfa7ab03dca7e63ee3825f4100e82b1433421a3804c45a9e93308bc1e44e"}
Sep 17 15:54:19.029111 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:19.029065    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ht2p6" podStartSLOduration=6.560256587 podStartE2EDuration="24.029050756s" totalImagesPullingTime="17.468794169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:19.028358998 +0000 UTC m=+25.293874001" watchObservedRunningTime="2026-09-17 15:54:19.029050756 +0000 UTC m=+25.294565757"
Sep 17 15:54:19.777276 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:19.777195    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:19.777276 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:19.777238    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:19.777276 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:19.777193    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:19.777563 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:19.777336    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:19.777563 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:19.777469    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:19.777681 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:19.777564    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:20.017042 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:20.016992    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"f060547929de15f4bb8977fd8e4d881ec65f3e07247bda2d44689714d4df34bc"}
Sep 17 15:54:20.983387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:20.983353    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:54:20.984029 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:20.984007    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:54:21.007245 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:21.007191    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rghxj" podStartSLOduration=5.929082536 podStartE2EDuration="25.007173315s" totalImagesPullingTime="19.078090779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:19.055614308 +0000 UTC m=+25.321129308" watchObservedRunningTime="2026-09-17 15:54:21.007173315 +0000 UTC m=+27.272688317"
Sep 17 15:54:21.019871 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:21.019838    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:54:21.020311 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:21.020250    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s6qlb"
Sep 17 15:54:21.777539 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:21.777514    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:21.777705 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:21.777544    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:21.777705 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:21.777514    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:21.777705 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:21.777636    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:21.777864 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:21.777778    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:21.777864 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:21.777852    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:23.027471 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:23.027276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" event={"ID":"bb38688f-9e3c-4245-b721-c31842c74f70","Type":"ContainerStarted","Data":"b65cc7c1cea4c1ac20db07f252d6723b037c8fb84b7cc033e88ab7bb4e6e6256"}
Sep 17 15:54:23.063278 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:23.063231    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj" podStartSLOduration=9.322464851 podStartE2EDuration="27.063216623s" totalImagesPullingTime="17.740751772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:23.062082852 +0000 UTC m=+29.327597852" watchObservedRunningTime="2026-09-17 15:54:23.063216623 +0000 UTC m=+29.328731619"
Sep 17 15:54:23.777270 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:23.777239    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:23.777426 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:23.777303    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:23.777426 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:23.777247    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:23.777426 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:23.777395    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:23.777542 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:23.777506    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:23.777599 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:23.777585    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:24.033067 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.032971    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6c09c6b-33dd-4de9-aab8-222a50919654" containerID="eb19297be6f704126c2d318de055f9ff2e4a0f3e090307d0fcacd7c867d272f3" exitCode=0
Sep 17 15:54:24.033891 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.033066    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s28wz" event={"ID":"d6c09c6b-33dd-4de9-aab8-222a50919654","Type":"ContainerDied","Data":"eb19297be6f704126c2d318de055f9ff2e4a0f3e090307d0fcacd7c867d272f3"}
Sep 17 15:54:24.033891 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.033608    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:54:24.033891 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.033698    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:54:24.033891 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.033751    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:54:24.053089 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.053059    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:54:24.053777 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.053756    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:54:24.748308 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.748153    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xh8xc"]
Sep 17 15:54:24.748308 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.748277    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:24.748508 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:24.748389    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:24.752971 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.752947    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-29lpf"]
Sep 17 15:54:24.753082 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.753029    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:24.753117 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:24.753098    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:24.758561 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.758536    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q4f6r"]
Sep 17 15:54:24.758650 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:24.758620    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:24.758738 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:24.758718    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:26.039805 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:26.039769    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6c09c6b-33dd-4de9-aab8-222a50919654" containerID="7d174c01cbbc36f31811a52eb85757ec766e1115fdc837c0281ce24e0a94f048" exitCode=0
Sep 17 15:54:26.040231 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:26.039849    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s28wz" event={"ID":"d6c09c6b-33dd-4de9-aab8-222a50919654","Type":"ContainerDied","Data":"7d174c01cbbc36f31811a52eb85757ec766e1115fdc837c0281ce24e0a94f048"}
Sep 17 15:54:26.776833 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:26.776808    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:26.776972 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:26.776808    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:26.777030 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:26.776809    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:26.777030 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:26.776975    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:26.777100 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:26.776896    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:26.777100 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:26.777055    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:28.048770 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:28.048728    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6c09c6b-33dd-4de9-aab8-222a50919654" containerID="5e46d871fac97471043f2e9fba4025147a94f1b9cb110eda893fecf6047a4cb0" exitCode=0
Sep 17 15:54:28.049222 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:28.048783    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s28wz" event={"ID":"d6c09c6b-33dd-4de9-aab8-222a50919654","Type":"ContainerDied","Data":"5e46d871fac97471043f2e9fba4025147a94f1b9cb110eda893fecf6047a4cb0"}
Sep 17 15:54:28.776710 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:28.776677    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:28.776913 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:28.776689    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:28.776913 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:28.776794    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-29lpf" podUID="d398cd52-c7fe-4df9-832c-1c0a2c357e45"
Sep 17 15:54:28.776913 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:28.776880    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xh8xc" podUID="afc9f293-dabb-4bff-b819-41dc8625ce82"
Sep 17 15:54:28.776913 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:28.776689    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:28.777113 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:28.776963    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q4f6r" podUID="0f843638-5eea-4ff9-99e4-33229d7815a3"
Sep 17 15:54:29.426994 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:29.426968    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:29.427388 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:29.427002    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:29.427388 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:29.427122    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:29.427388 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:29.427202    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret podName:0f843638-5eea-4ff9-99e4-33229d7815a3 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:01.427185947 +0000 UTC m=+67.692700925 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret") pod "global-pull-secret-syncer-q4f6r" (UID: "0f843638-5eea-4ff9-99e4-33229d7815a3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:29.427388 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:29.427127    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:29.427388 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:29.427265    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs podName:afc9f293-dabb-4bff-b819-41dc8625ce82 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:01.427250342 +0000 UTC m=+67.692765321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs") pod "network-metrics-daemon-xh8xc" (UID: "afc9f293-dabb-4bff-b819-41dc8625ce82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:29.527843 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:29.527808    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:29.527997 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:29.527944    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:29.527997 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:29.527969    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:29.527997 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:29.527983    2585 projected.go:196] Error preparing data for projected volume kube-api-access-kj4bl for pod openshift-network-diagnostics/network-check-target-29lpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:29.528111 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:29.528043    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl podName:d398cd52-c7fe-4df9-832c-1c0a2c357e45 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:01.528025757 +0000 UTC m=+67.793540739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj4bl" (UniqueName: "kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl") pod "network-check-target-29lpf" (UID: "d398cd52-c7fe-4df9-832c-1c0a2c357e45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:29.797740 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:29.797714    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-18.ec2.internal" event="NodeReady"
Sep 17 15:54:29.797915 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:29.797830    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:54:29.987985 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:29.987953    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-579d78b6fc-lbj92"]
Sep 17 15:54:30.009673 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.009640    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.017181 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.017154    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:54:30.017617 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.017451    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:54:30.017617 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.017470    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:54:30.017617 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.017509    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qw6lt\""
Sep 17 15:54:30.029256 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.029233    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579d78b6fc-lbj92"]
Sep 17 15:54:30.039010 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.038991    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:54:30.061649 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.061591    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dbb7d"]
Sep 17 15:54:30.082821 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.082802    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v5ms8"]
Sep 17 15:54:30.082961 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.082943    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.088982 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.088957    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:54:30.095760 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.095743    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:54:30.096920 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.096839    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcdnd\""
Sep 17 15:54:30.107439 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.107414    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbb7d"]
Sep 17 15:54:30.107565 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.107467    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v5ms8"
Sep 17 15:54:30.107565 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.107474    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v5ms8"]
Sep 17 15:54:30.114035 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.114015    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:54:30.114134 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.114076    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:54:30.114225 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.114208    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:54:30.127257 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.127234    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j6zt9\""
Sep 17 15:54:30.132035 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.132010    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-installation-pull-secrets\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.132123 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.132055    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-image-registry-private-configuration\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.132174 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.132119    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqmx\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-kube-api-access-ngqmx\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.132174 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.132143    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.132174 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.132169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-certificates\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.132327 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.132190    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-bound-sa-token\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.132327 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.132217    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-trusted-ca\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.132327 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.132244    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d416fdcc-1345-4037-b055-304ba1f0b3a5-ca-trust-extracted\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.232972 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.232938    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-installation-pull-secrets\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.233168 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.232980    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-image-registry-private-configuration\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.233168 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233016    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbmq\" (UniqueName: \"kubernetes.io/projected/48fc9971-ac9e-4939-853c-cb4851a1f652-kube-api-access-bsbmq\") pod \"ingress-canary-v5ms8\" (UID: \"48fc9971-ac9e-4939-853c-cb4851a1f652\") " pod="openshift-ingress-canary/ingress-canary-v5ms8"
Sep 17 15:54:30.233168 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233042    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49b31cc2-74bc-48f2-8420-9bba8de27ab5-config-volume\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.233168 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.233168 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233126    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqmx\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-kube-api-access-ngqmx\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.233168 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233149    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mp8\" (UniqueName: \"kubernetes.io/projected/49b31cc2-74bc-48f2-8420-9bba8de27ab5-kube-api-access-r7mp8\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.233429 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233175    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.233429 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233204    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-certificates\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.233429 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233226    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-bound-sa-token\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.233429 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233247    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48fc9971-ac9e-4939-853c-cb4851a1f652-cert\") pod \"ingress-canary-v5ms8\" (UID: \"48fc9971-ac9e-4939-853c-cb4851a1f652\") " pod="openshift-ingress-canary/ingress-canary-v5ms8"
Sep 17 15:54:30.233429 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233272    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-trusted-ca\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.233429 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233313    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b31cc2-74bc-48f2-8420-9bba8de27ab5-tmp-dir\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.233429 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233340    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d416fdcc-1345-4037-b055-304ba1f0b3a5-ca-trust-extracted\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.233810 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.233783    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d416fdcc-1345-4037-b055-304ba1f0b3a5-ca-trust-extracted\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.234054 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.234028    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:30.234054 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.234053    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d78b6fc-lbj92: secret "image-registry-tls" not found
Sep 17 15:54:30.234206 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.234123    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls podName:d416fdcc-1345-4037-b055-304ba1f0b3a5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:30.734103 +0000 UTC m=+36.999618005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls") pod "image-registry-579d78b6fc-lbj92" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5") : secret "image-registry-tls" not found
Sep 17 15:54:30.234265 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.234217    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-certificates\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.234931 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.234910    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-trusted-ca\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.237706 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.237595    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-installation-pull-secrets\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.237789 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.237621    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-image-registry-private-configuration\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.254922 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.254894    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-bound-sa-token\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.265584 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.265559    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqmx\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-kube-api-access-ngqmx\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.334106 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.334034    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbmq\" (UniqueName: \"kubernetes.io/projected/48fc9971-ac9e-4939-853c-cb4851a1f652-kube-api-access-bsbmq\") pod \"ingress-canary-v5ms8\" (UID: \"48fc9971-ac9e-4939-853c-cb4851a1f652\") " pod="openshift-ingress-canary/ingress-canary-v5ms8"
Sep 17 15:54:30.334106 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.334065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49b31cc2-74bc-48f2-8420-9bba8de27ab5-config-volume\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.334106 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.334079    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.334329 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.334129    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mp8\" (UniqueName: \"kubernetes.io/projected/49b31cc2-74bc-48f2-8420-9bba8de27ab5-kube-api-access-r7mp8\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.334329 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.334208    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:30.334329 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.334299    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48fc9971-ac9e-4939-853c-cb4851a1f652-cert\") pod \"ingress-canary-v5ms8\" (UID: \"48fc9971-ac9e-4939-853c-cb4851a1f652\") " pod="openshift-ingress-canary/ingress-canary-v5ms8"
Sep 17 15:54:30.334473 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.334345    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b31cc2-74bc-48f2-8420-9bba8de27ab5-tmp-dir\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.334473 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.334385    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls podName:49b31cc2-74bc-48f2-8420-9bba8de27ab5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:30.834364019 +0000 UTC m=+37.099879000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls") pod "dns-default-dbb7d" (UID: "49b31cc2-74bc-48f2-8420-9bba8de27ab5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:30.334615 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.334598    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b31cc2-74bc-48f2-8420-9bba8de27ab5-tmp-dir\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.334705 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.334683    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49b31cc2-74bc-48f2-8420-9bba8de27ab5-config-volume\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.336627 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.336609    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48fc9971-ac9e-4939-853c-cb4851a1f652-cert\") pod \"ingress-canary-v5ms8\" (UID: \"48fc9971-ac9e-4939-853c-cb4851a1f652\") " pod="openshift-ingress-canary/ingress-canary-v5ms8"
Sep 17 15:54:30.346202 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.346184    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbmq\" (UniqueName: \"kubernetes.io/projected/48fc9971-ac9e-4939-853c-cb4851a1f652-kube-api-access-bsbmq\") pod \"ingress-canary-v5ms8\" (UID: \"48fc9971-ac9e-4939-853c-cb4851a1f652\") " pod="openshift-ingress-canary/ingress-canary-v5ms8"
Sep 17 15:54:30.351170 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.351153    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mp8\" (UniqueName: \"kubernetes.io/projected/49b31cc2-74bc-48f2-8420-9bba8de27ab5-kube-api-access-r7mp8\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.417787 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.417762    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v5ms8"
Sep 17 15:54:30.737196 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.737113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:30.737746 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.737278    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:30.737746 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.737317    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d78b6fc-lbj92: secret "image-registry-tls" not found
Sep 17 15:54:30.737746 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.737388    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls podName:d416fdcc-1345-4037-b055-304ba1f0b3a5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:31.737366495 +0000 UTC m=+38.002881495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls") pod "image-registry-579d78b6fc-lbj92" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5") : secret "image-registry-tls" not found
Sep 17 15:54:30.777307 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.777251    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:54:30.777307 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.777255    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:54:30.777553 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.777381    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:54:30.791504 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.791473    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9smtj\""
Sep 17 15:54:30.791629 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.791592    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:54:30.791693 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.791474    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:54:30.791902 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.791884    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:54:30.792043 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.791916    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:54:30.792246 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.792215    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hxr8k\""
Sep 17 15:54:30.804254 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.804234    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v5ms8"]
Sep 17 15:54:30.820114 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:54:30.820091    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fc9971_ac9e_4939_853c_cb4851a1f652.slice/crio-67cde9d90c2c6306691f5a3231ee7e3dd056cb246411ed2f52a654857b4a671d WatchSource:0}: Error finding container 67cde9d90c2c6306691f5a3231ee7e3dd056cb246411ed2f52a654857b4a671d: Status 404 returned error can't find the container with id 67cde9d90c2c6306691f5a3231ee7e3dd056cb246411ed2f52a654857b4a671d
Sep 17 15:54:30.837643 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:30.837618    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:30.837802 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.837780    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:30.837898 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:30.837849    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls podName:49b31cc2-74bc-48f2-8420-9bba8de27ab5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:31.837830748 +0000 UTC m=+38.103345743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls") pod "dns-default-dbb7d" (UID: "49b31cc2-74bc-48f2-8420-9bba8de27ab5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:31.057790 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:31.057750    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v5ms8" event={"ID":"48fc9971-ac9e-4939-853c-cb4851a1f652","Type":"ContainerStarted","Data":"67cde9d90c2c6306691f5a3231ee7e3dd056cb246411ed2f52a654857b4a671d"}
Sep 17 15:54:31.744190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:31.744147    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:31.744783 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:31.744337    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:31.744783 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:31.744359    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d78b6fc-lbj92: secret "image-registry-tls" not found
Sep 17 15:54:31.744783 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:31.744430    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls podName:d416fdcc-1345-4037-b055-304ba1f0b3a5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:33.744410254 +0000 UTC m=+40.009925251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls") pod "image-registry-579d78b6fc-lbj92" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5") : secret "image-registry-tls" not found
Sep 17 15:54:31.845082 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:31.845042    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:31.845265 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:31.845217    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:31.845351 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:31.845276    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls podName:49b31cc2-74bc-48f2-8420-9bba8de27ab5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:33.845259415 +0000 UTC m=+40.110774418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls") pod "dns-default-dbb7d" (UID: "49b31cc2-74bc-48f2-8420-9bba8de27ab5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:33.760669 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:33.760630    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:33.761221 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:33.760770    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:33.761221 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:33.760788    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d78b6fc-lbj92: secret "image-registry-tls" not found
Sep 17 15:54:33.761221 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:33.760840    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls podName:d416fdcc-1345-4037-b055-304ba1f0b3a5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:37.760827232 +0000 UTC m=+44.026342223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls") pod "image-registry-579d78b6fc-lbj92" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5") : secret "image-registry-tls" not found
Sep 17 15:54:33.861935 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:33.861772    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:33.862084 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:33.861981    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:33.862084 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:33.862058    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls podName:49b31cc2-74bc-48f2-8420-9bba8de27ab5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:37.862036267 +0000 UTC m=+44.127551249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls") pod "dns-default-dbb7d" (UID: "49b31cc2-74bc-48f2-8420-9bba8de27ab5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:35.071609 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:35.071426    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6c09c6b-33dd-4de9-aab8-222a50919654" containerID="7135b8cb6d303eb782f0af2bd08a27bf0492548de326dce1807b77e409b3792a" exitCode=0
Sep 17 15:54:35.072024 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:35.071496    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s28wz" event={"ID":"d6c09c6b-33dd-4de9-aab8-222a50919654","Type":"ContainerDied","Data":"7135b8cb6d303eb782f0af2bd08a27bf0492548de326dce1807b77e409b3792a"}
Sep 17 15:54:35.073261 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:35.073235    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v5ms8" event={"ID":"48fc9971-ac9e-4939-853c-cb4851a1f652","Type":"ContainerStarted","Data":"3d1bd902773b9df0bd9afbd53d7fd6ed132354b3f6785107fce54774c4365567"}
Sep 17 15:54:35.128944 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:35.127920    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v5ms8" podStartSLOduration=1.439776811 podStartE2EDuration="5.127896928s" totalImagesPullingTime="3.688120117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:35.127702135 +0000 UTC m=+41.393217136" watchObservedRunningTime="2026-09-17 15:54:35.127896928 +0000 UTC m=+41.393411929"
Sep 17 15:54:36.079208 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:36.079175    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6c09c6b-33dd-4de9-aab8-222a50919654" containerID="26948c0237d5efa40371fe3a45caf8f9d98765f28077b93c20d163e0d5622ee4" exitCode=0
Sep 17 15:54:36.079627 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:36.079262    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s28wz" event={"ID":"d6c09c6b-33dd-4de9-aab8-222a50919654","Type":"ContainerDied","Data":"26948c0237d5efa40371fe3a45caf8f9d98765f28077b93c20d163e0d5622ee4"}
Sep 17 15:54:37.085499 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:37.085461    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s28wz" event={"ID":"d6c09c6b-33dd-4de9-aab8-222a50919654","Type":"ContainerStarted","Data":"40efaa8c7cec42a687a68aafa488c77d082fa76b032057bdd3b09f9db596a611"}
Sep 17 15:54:37.222683 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:37.222624    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s28wz" podStartSLOduration=9.466919856 podStartE2EDuration="41.222607792s" totalImagesPullingTime="31.755687936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:37.213667416 +0000 UTC m=+43.479182416" watchObservedRunningTime="2026-09-17 15:54:37.222607792 +0000 UTC m=+43.488122791"
Sep 17 15:54:37.787936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:37.787896    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:37.788176 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:37.788013    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:37.788176 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:37.788026    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d78b6fc-lbj92: secret "image-registry-tls" not found
Sep 17 15:54:37.788176 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:37.788080    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls podName:d416fdcc-1345-4037-b055-304ba1f0b3a5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:45.788062219 +0000 UTC m=+52.053577217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls") pod "image-registry-579d78b6fc-lbj92" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5") : secret "image-registry-tls" not found
Sep 17 15:54:37.888969 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:37.888915    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:37.889184 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:37.889096    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:37.889184 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:37.889179    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls podName:49b31cc2-74bc-48f2-8420-9bba8de27ab5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:45.889157869 +0000 UTC m=+52.154672865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls") pod "dns-default-dbb7d" (UID: "49b31cc2-74bc-48f2-8420-9bba8de27ab5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:45.849613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:45.849570    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:54:45.849990 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:45.849686    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:45.849990 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:45.849698    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d78b6fc-lbj92: secret "image-registry-tls" not found
Sep 17 15:54:45.849990 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:45.849750    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls podName:d416fdcc-1345-4037-b055-304ba1f0b3a5 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:01.849735578 +0000 UTC m=+68.115250569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls") pod "image-registry-579d78b6fc-lbj92" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5") : secret "image-registry-tls" not found
Sep 17 15:54:45.950229 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:45.950187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:54:45.950426 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:45.950358    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:45.950426 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:54:45.950418    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls podName:49b31cc2-74bc-48f2-8420-9bba8de27ab5 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:01.950400126 +0000 UTC m=+68.215915112 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls") pod "dns-default-dbb7d" (UID: "49b31cc2-74bc-48f2-8420-9bba8de27ab5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:56.054416 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:54:56.054388    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ttzj"
Sep 17 15:55:01.460702 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.460640    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:55:01.460702 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.460698    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:55:01.464137 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.464114    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:55:01.464256 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.464116    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:55:01.471827 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:01.471804    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:55:01.471913 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:01.471883    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs podName:afc9f293-dabb-4bff-b819-41dc8625ce82 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:05.471862311 +0000 UTC m=+131.737377311 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs") pod "network-metrics-daemon-xh8xc" (UID: "afc9f293-dabb-4bff-b819-41dc8625ce82") : secret "metrics-daemon-secret" not found
Sep 17 15:55:01.474422 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.474399    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f843638-5eea-4ff9-99e4-33229d7815a3-original-pull-secret\") pod \"global-pull-secret-syncer-q4f6r\" (UID: \"0f843638-5eea-4ff9-99e4-33229d7815a3\") " pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:55:01.561408 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.561368    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:55:01.564647 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.564632    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:55:01.574463 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.574443    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:55:01.585243 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.585222    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4bl\" (UniqueName: \"kubernetes.io/projected/d398cd52-c7fe-4df9-832c-1c0a2c357e45-kube-api-access-kj4bl\") pod \"network-check-target-29lpf\" (UID: \"d398cd52-c7fe-4df9-832c-1c0a2c357e45\") " pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:55:01.700110 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.700084    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hxr8k\""
Sep 17 15:55:01.703196 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.703179    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q4f6r"
Sep 17 15:55:01.706804 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.706789    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:55:01.863131 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.863104    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:55:01.863333 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:01.863267    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:01.863333 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:01.863310    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d78b6fc-lbj92: secret "image-registry-tls" not found
Sep 17 15:55:01.863411 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:01.863372    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls podName:d416fdcc-1345-4037-b055-304ba1f0b3a5 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:33.863354818 +0000 UTC m=+100.128869809 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls") pod "image-registry-579d78b6fc-lbj92" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5") : secret "image-registry-tls" not found
Sep 17 15:55:01.963557 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.963511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:55:01.963709 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:01.963631    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:55:01.963709 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:01.963697    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls podName:49b31cc2-74bc-48f2-8420-9bba8de27ab5 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:33.963679249 +0000 UTC m=+100.229194242 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls") pod "dns-default-dbb7d" (UID: "49b31cc2-74bc-48f2-8420-9bba8de27ab5") : secret "dns-default-metrics-tls" not found
Sep 17 15:55:01.984807 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.984724    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q4f6r"]
Sep 17 15:55:01.987101 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:01.987076    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-29lpf"]
Sep 17 15:55:01.988417 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:55:01.988398    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f843638_5eea_4ff9_99e4_33229d7815a3.slice/crio-e66d8ca82dee4ab4ad89b8a1bd9f8fb9e81d8e87b350f4c815f583c44dd393fb WatchSource:0}: Error finding container e66d8ca82dee4ab4ad89b8a1bd9f8fb9e81d8e87b350f4c815f583c44dd393fb: Status 404 returned error can't find the container with id e66d8ca82dee4ab4ad89b8a1bd9f8fb9e81d8e87b350f4c815f583c44dd393fb
Sep 17 15:55:01.990151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:55:01.990093    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd398cd52_c7fe_4df9_832c_1c0a2c357e45.slice/crio-f3f32fca10d556c46d336756903edb85c3fa7da1f60fdecf5698eb20c46391e5 WatchSource:0}: Error finding container f3f32fca10d556c46d336756903edb85c3fa7da1f60fdecf5698eb20c46391e5: Status 404 returned error can't find the container with id f3f32fca10d556c46d336756903edb85c3fa7da1f60fdecf5698eb20c46391e5
Sep 17 15:55:02.149502 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:02.149462    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-29lpf" event={"ID":"d398cd52-c7fe-4df9-832c-1c0a2c357e45","Type":"ContainerStarted","Data":"f3f32fca10d556c46d336756903edb85c3fa7da1f60fdecf5698eb20c46391e5"}
Sep 17 15:55:02.150601 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:02.150579    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q4f6r" event={"ID":"0f843638-5eea-4ff9-99e4-33229d7815a3","Type":"ContainerStarted","Data":"e66d8ca82dee4ab4ad89b8a1bd9f8fb9e81d8e87b350f4c815f583c44dd393fb"}
Sep 17 15:55:07.166686 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:07.166644    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-29lpf" event={"ID":"d398cd52-c7fe-4df9-832c-1c0a2c357e45","Type":"ContainerStarted","Data":"43cd74381dc97f4bbae9af235c8d7f55f2f7d8081877ecc2c3d1c574b9db0330"}
Sep 17 15:55:07.167170 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:07.166767    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:55:07.168228 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:07.168207    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q4f6r" event={"ID":"0f843638-5eea-4ff9-99e4-33229d7815a3","Type":"ContainerStarted","Data":"482614f08f5edc430951586fa8cbd357b789a69ee48c280664a2a70e2c57a552"}
Sep 17 15:55:07.188736 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:07.188685    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-29lpf" podStartSLOduration=66.492806973 podStartE2EDuration="1m11.188668222s" totalImagesPullingTime="4.695861249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:07.187791991 +0000 UTC m=+73.453306990" watchObservedRunningTime="2026-09-17 15:55:07.188668222 +0000 UTC m=+73.454183222"
Sep 17 15:55:07.207066 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:07.207011    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q4f6r" podStartSLOduration=66.505762115 podStartE2EDuration="1m11.206994947s" totalImagesPullingTime="4.701232832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:07.206175128 +0000 UTC m=+73.471690127" watchObservedRunningTime="2026-09-17 15:55:07.206994947 +0000 UTC m=+73.472509946"
Sep 17 15:55:22.977105 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:22.977068    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-l7rxh"]
Sep 17 15:55:22.982000 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:22.981976    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:22.989992 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:22.989966    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 15:55:22.990118 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:22.989966    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:22.990118 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:22.990040    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:23.002100 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.002069    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q8ctc\""
Sep 17 15:55:23.002418 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.002403    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 15:55:23.003807 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.003791    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 15:55:23.010857 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.010834    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/53c5829f-704f-4409-a7f6-95ac75a37a45-trusted-ca\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.010973 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.010904    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53c5829f-704f-4409-a7f6-95ac75a37a45-config\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.010973 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.010962    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53c5829f-704f-4409-a7f6-95ac75a37a45-serving-cert\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.011040 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.010988    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjff\" (UniqueName: \"kubernetes.io/projected/53c5829f-704f-4409-a7f6-95ac75a37a45-kube-api-access-jvjff\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.019053 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.019032    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-l7rxh"]
Sep 17 15:55:23.056639 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.056606    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"]
Sep 17 15:55:23.059456 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.059439    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:23.063215 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.063191    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 15:55:23.063329 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.063243    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-72d9q\""
Sep 17 15:55:23.063597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.063582    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 15:55:23.076241 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.076214    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"]
Sep 17 15:55:23.112203 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.112158    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/53c5829f-704f-4409-a7f6-95ac75a37a45-trusted-ca\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.112203 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.112208    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/498bb02c-2602-4a5b-807d-bf5e8ad38c82-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:23.112501 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.112262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53c5829f-704f-4409-a7f6-95ac75a37a45-config\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.112501 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.112310    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53c5829f-704f-4409-a7f6-95ac75a37a45-serving-cert\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.112501 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.112374    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:23.112501 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.112409    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjff\" (UniqueName: \"kubernetes.io/projected/53c5829f-704f-4409-a7f6-95ac75a37a45-kube-api-access-jvjff\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.113042 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.113017    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53c5829f-704f-4409-a7f6-95ac75a37a45-config\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.113226 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.113204    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/53c5829f-704f-4409-a7f6-95ac75a37a45-trusted-ca\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.114749 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.114732    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53c5829f-704f-4409-a7f6-95ac75a37a45-serving-cert\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.123096 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.123077    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjff\" (UniqueName: \"kubernetes.io/projected/53c5829f-704f-4409-a7f6-95ac75a37a45-kube-api-access-jvjff\") pod \"console-operator-fd4bb46c-l7rxh\" (UID: \"53c5829f-704f-4409-a7f6-95ac75a37a45\") " pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.161370 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.161334    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"]
Sep 17 15:55:23.164453 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.164435    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.168947 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.168918    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:23.169105 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.168969    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 15:55:23.169105 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.168983    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 15:55:23.169105 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.168978    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:23.169105 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.169036    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cx4fj\""
Sep 17 15:55:23.185594 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.185561    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"]
Sep 17 15:55:23.213403 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.213366    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:23.213403 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.213406    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/452444d7-ed37-4ddb-b83a-192ee0ef5727-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.213635 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.213446    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrmg\" (UniqueName: \"kubernetes.io/projected/452444d7-ed37-4ddb-b83a-192ee0ef5727-kube-api-access-jbrmg\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.213635 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.213514    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/498bb02c-2602-4a5b-807d-bf5e8ad38c82-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:23.213635 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:23.213529    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 15:55:23.213635 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:23.213603    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert podName:498bb02c-2602-4a5b-807d-bf5e8ad38c82 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:23.713581802 +0000 UTC m=+89.979096812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dpcws" (UID: "498bb02c-2602-4a5b-807d-bf5e8ad38c82") : secret "networking-console-plugin-cert" not found
Sep 17 15:55:23.213816 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.213649    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/452444d7-ed37-4ddb-b83a-192ee0ef5727-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.214148 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.214127    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/498bb02c-2602-4a5b-807d-bf5e8ad38c82-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:23.290733 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.290697    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:23.315072 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.315027    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/452444d7-ed37-4ddb-b83a-192ee0ef5727-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.315254 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.315083    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/452444d7-ed37-4ddb-b83a-192ee0ef5727-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.315254 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.315112    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrmg\" (UniqueName: \"kubernetes.io/projected/452444d7-ed37-4ddb-b83a-192ee0ef5727-kube-api-access-jbrmg\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.315750 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.315724    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/452444d7-ed37-4ddb-b83a-192ee0ef5727-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.317584 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.317560    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/452444d7-ed37-4ddb-b83a-192ee0ef5727-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.326819 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.326786    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrmg\" (UniqueName: \"kubernetes.io/projected/452444d7-ed37-4ddb-b83a-192ee0ef5727-kube-api-access-jbrmg\") pod \"kube-storage-version-migrator-operator-55889f7b47-ts59f\" (UID: \"452444d7-ed37-4ddb-b83a-192ee0ef5727\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.473276 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.473215    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"
Sep 17 15:55:23.568175 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.568096    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-l7rxh"]
Sep 17 15:55:23.571191 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:55:23.571158    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c5829f_704f_4409_a7f6_95ac75a37a45.slice/crio-68d5e8ef59054bc75b220ea236a47d0cb184f4eb5ac71beb07efa075fac4a0e2 WatchSource:0}: Error finding container 68d5e8ef59054bc75b220ea236a47d0cb184f4eb5ac71beb07efa075fac4a0e2: Status 404 returned error can't find the container with id 68d5e8ef59054bc75b220ea236a47d0cb184f4eb5ac71beb07efa075fac4a0e2
Sep 17 15:55:23.718566 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.718527    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:23.718753 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:23.718669    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 15:55:23.718753 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:23.718735    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert podName:498bb02c-2602-4a5b-807d-bf5e8ad38c82 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:24.718720042 +0000 UTC m=+90.984235020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dpcws" (UID: "498bb02c-2602-4a5b-807d-bf5e8ad38c82") : secret "networking-console-plugin-cert" not found
Sep 17 15:55:23.748670 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:23.748632    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f"]
Sep 17 15:55:23.753180 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:55:23.753152    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452444d7_ed37_4ddb_b83a_192ee0ef5727.slice/crio-66de5432209ae59a911cf45a31c438121eb7a9fe60f1163642298d3c96be110b WatchSource:0}: Error finding container 66de5432209ae59a911cf45a31c438121eb7a9fe60f1163642298d3c96be110b: Status 404 returned error can't find the container with id 66de5432209ae59a911cf45a31c438121eb7a9fe60f1163642298d3c96be110b
Sep 17 15:55:24.212521 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:24.212473    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" event={"ID":"53c5829f-704f-4409-a7f6-95ac75a37a45","Type":"ContainerStarted","Data":"68d5e8ef59054bc75b220ea236a47d0cb184f4eb5ac71beb07efa075fac4a0e2"}
Sep 17 15:55:24.213838 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:24.213802    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f" event={"ID":"452444d7-ed37-4ddb-b83a-192ee0ef5727","Type":"ContainerStarted","Data":"66de5432209ae59a911cf45a31c438121eb7a9fe60f1163642298d3c96be110b"}
Sep 17 15:55:24.727211 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:24.727170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:24.727429 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:24.727345    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 15:55:24.727499 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:24.727436    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert podName:498bb02c-2602-4a5b-807d-bf5e8ad38c82 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:26.727413643 +0000 UTC m=+92.992928636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dpcws" (UID: "498bb02c-2602-4a5b-807d-bf5e8ad38c82") : secret "networking-console-plugin-cert" not found
Sep 17 15:55:26.222055 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.221949    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/0.log"
Sep 17 15:55:26.222055 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.222027    2585 generic.go:356] "Generic (PLEG): container finished" podID="53c5829f-704f-4409-a7f6-95ac75a37a45" containerID="f26255c256a2bd33f0e70a959c8700848fe94bc86924c7137597364b0865b634" exitCode=255
Sep 17 15:55:26.222551 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.222124    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" event={"ID":"53c5829f-704f-4409-a7f6-95ac75a37a45","Type":"ContainerDied","Data":"f26255c256a2bd33f0e70a959c8700848fe94bc86924c7137597364b0865b634"}
Sep 17 15:55:26.222551 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.222361    2585 scope.go:117] "RemoveContainer" containerID="f26255c256a2bd33f0e70a959c8700848fe94bc86924c7137597364b0865b634"
Sep 17 15:55:26.223960 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.223930    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f" event={"ID":"452444d7-ed37-4ddb-b83a-192ee0ef5727","Type":"ContainerStarted","Data":"9348b75a8e4d293596aa1431141f89d12f977be8cf06662799279cf09388ead2"}
Sep 17 15:55:26.264874 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.264817    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f" podStartSLOduration=1.199391915 podStartE2EDuration="3.264795872s" totalImagesPullingTime="2.065403957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:26.263713762 +0000 UTC m=+92.529228762" watchObservedRunningTime="2026-09-17 15:55:26.264795872 +0000 UTC m=+92.530310908"
Sep 17 15:55:26.745861 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.745813    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:26.746083 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:26.745998    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 15:55:26.746153 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:26.746087    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert podName:498bb02c-2602-4a5b-807d-bf5e8ad38c82 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.746064407 +0000 UTC m=+97.011579404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dpcws" (UID: "498bb02c-2602-4a5b-807d-bf5e8ad38c82") : secret "networking-console-plugin-cert" not found
Sep 17 15:55:26.873911 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.873865    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5mrxm"]
Sep 17 15:55:26.877313 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.877272    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:26.881380 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.881355    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9ppb\""
Sep 17 15:55:26.882141 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.882119    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:55:26.882141 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.882137    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:55:26.882402 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.882121    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:55:26.882402 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.882386    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:55:26.898521 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.898485    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5mrxm"]
Sep 17 15:55:26.947532 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.947490    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-data-volume\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:26.947532 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.947538    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:26.947768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.947637    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:26.947768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.947666    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-crio-socket\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:26.947768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:26.947693    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4lw\" (UniqueName: \"kubernetes.io/projected/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-api-access-9n4lw\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.048844 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.048804    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.048844 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.048847    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-crio-socket\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.049073 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.048884    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4lw\" (UniqueName: \"kubernetes.io/projected/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-api-access-9n4lw\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.049073 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.048947    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:27.049073 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.048973    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-data-volume\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.049073 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.049000    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.049073 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.048954    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-crio-socket\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.049073 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.049040    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:27.549020555 +0000 UTC m=+93.814535577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:27.049263 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.049107    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:27.049263 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.049153    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:27.549138033 +0000 UTC m=+93.814653027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:27.049386 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.049369    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-data-volume\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.060912 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.060884    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4lw\" (UniqueName: \"kubernetes.io/projected/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-api-access-9n4lw\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.229091 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.229062    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/1.log"
Sep 17 15:55:27.229643 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.229587    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/0.log"
Sep 17 15:55:27.229643 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.229627    2585 generic.go:356] "Generic (PLEG): container finished" podID="53c5829f-704f-4409-a7f6-95ac75a37a45" containerID="1d093dfe283727be95adeabe3f981f92f6251f60be196ae08154fd1ac6b4fdb1" exitCode=255
Sep 17 15:55:27.229739 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.229715    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" event={"ID":"53c5829f-704f-4409-a7f6-95ac75a37a45","Type":"ContainerDied","Data":"1d093dfe283727be95adeabe3f981f92f6251f60be196ae08154fd1ac6b4fdb1"}
Sep 17 15:55:27.229793 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.229780    2585 scope.go:117] "RemoveContainer" containerID="f26255c256a2bd33f0e70a959c8700848fe94bc86924c7137597364b0865b634"
Sep 17 15:55:27.230063 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.230037    2585 scope.go:117] "RemoveContainer" containerID="1d093dfe283727be95adeabe3f981f92f6251f60be196ae08154fd1ac6b4fdb1"
Sep 17 15:55:27.230278 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.230250    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-l7rxh_openshift-console-operator(53c5829f-704f-4409-a7f6-95ac75a37a45)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" podUID="53c5829f-704f-4409-a7f6-95ac75a37a45"
Sep 17 15:55:27.552581 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.552532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.552788 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:27.552614    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:27.552788 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.552693    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:27.552788 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.552731    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:27.552788 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.552764    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:28.552748218 +0000 UTC m=+94.818263227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:27.552788 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:27.552786    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:28.552771101 +0000 UTC m=+94.818286101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:28.234385 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:28.234356    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/1.log"
Sep 17 15:55:28.234770 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:28.234675    2585 scope.go:117] "RemoveContainer" containerID="1d093dfe283727be95adeabe3f981f92f6251f60be196ae08154fd1ac6b4fdb1"
Sep 17 15:55:28.234837 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:28.234817    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-l7rxh_openshift-console-operator(53c5829f-704f-4409-a7f6-95ac75a37a45)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" podUID="53c5829f-704f-4409-a7f6-95ac75a37a45"
Sep 17 15:55:28.562333 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:28.562261    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:28.562520 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:28.562396    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:28.562520 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:28.562427    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:28.562520 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:28.562497    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:28.562520 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:28.562513    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.56249288 +0000 UTC m=+96.828008074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:28.562655 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:28.562535    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.562524521 +0000 UTC m=+96.828039523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:28.604859 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:28.604831    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ksfvw_64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb/dns-node-resolver/0.log"
Sep 17 15:55:30.000882 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.000851    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-prh25_74438d26-e6ab-4d96-a541-0f428f8951f2/node-ca/0.log"
Sep 17 15:55:30.287562 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.287484    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gz2xn"]
Sep 17 15:55:30.291714 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.291695    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.294734 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.294710    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 15:55:30.294872 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.294792    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mmbb6\""
Sep 17 15:55:30.295791 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.295769    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 15:55:30.295920 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.295830    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 15:55:30.295984 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.295933    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 15:55:30.304476 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.304452    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gz2xn"]
Sep 17 15:55:30.376949 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.376908    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdhxx\" (UniqueName: \"kubernetes.io/projected/019505ed-37a9-4981-b833-c6f88bb1a3ad-kube-api-access-cdhxx\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.377126 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.376990    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/019505ed-37a9-4981-b833-c6f88bb1a3ad-signing-cabundle\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.377126 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.377051    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/019505ed-37a9-4981-b833-c6f88bb1a3ad-signing-key\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.478356 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.478319    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/019505ed-37a9-4981-b833-c6f88bb1a3ad-signing-key\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.478483 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.478377    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdhxx\" (UniqueName: \"kubernetes.io/projected/019505ed-37a9-4981-b833-c6f88bb1a3ad-kube-api-access-cdhxx\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.478483 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.478422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/019505ed-37a9-4981-b833-c6f88bb1a3ad-signing-cabundle\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.478983 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.478964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/019505ed-37a9-4981-b833-c6f88bb1a3ad-signing-cabundle\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.481054 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.481031    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/019505ed-37a9-4981-b833-c6f88bb1a3ad-signing-key\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.488078 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.488055    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdhxx\" (UniqueName: \"kubernetes.io/projected/019505ed-37a9-4981-b833-c6f88bb1a3ad-kube-api-access-cdhxx\") pod \"service-ca-55f6fc7b86-gz2xn\" (UID: \"019505ed-37a9-4981-b833-c6f88bb1a3ad\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.579330 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.579216    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:30.579330 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.579305    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:30.579519 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:30.579381    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:30.579519 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:30.579393    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:30.579519 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:30.579441    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:34.579426107 +0000 UTC m=+100.844941105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:30.579519 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:30.579455    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:34.579448952 +0000 UTC m=+100.844963930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:30.600420 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.600382    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn"
Sep 17 15:55:30.781024 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.780992    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:30.781208 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:30.781102    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 15:55:30.781208 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:30.781152    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert podName:498bb02c-2602-4a5b-807d-bf5e8ad38c82 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:38.781138655 +0000 UTC m=+105.046653637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dpcws" (UID: "498bb02c-2602-4a5b-807d-bf5e8ad38c82") : secret "networking-console-plugin-cert" not found
Sep 17 15:55:30.800983 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.800957    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v5ms8_48fc9971-ac9e-4939-853c-cb4851a1f652/serve-healthcheck-canary/0.log"
Sep 17 15:55:30.873258 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:30.873177    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gz2xn"]
Sep 17 15:55:30.876469 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:55:30.876439    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019505ed_37a9_4981_b833_c6f88bb1a3ad.slice/crio-79afce5986c4a5e1e3ef17982384aca3f16be56faa5fe46a0da1da8e7eadb07c WatchSource:0}: Error finding container 79afce5986c4a5e1e3ef17982384aca3f16be56faa5fe46a0da1da8e7eadb07c: Status 404 returned error can't find the container with id 79afce5986c4a5e1e3ef17982384aca3f16be56faa5fe46a0da1da8e7eadb07c
Sep 17 15:55:31.243527 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:31.243450    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn" event={"ID":"019505ed-37a9-4981-b833-c6f88bb1a3ad","Type":"ContainerStarted","Data":"79afce5986c4a5e1e3ef17982384aca3f16be56faa5fe46a0da1da8e7eadb07c"}
Sep 17 15:55:33.250946 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:33.250916    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn" event={"ID":"019505ed-37a9-4981-b833-c6f88bb1a3ad","Type":"ContainerStarted","Data":"f5067b56db7ee359643193e4b4deb1d6d9f4c7d59bf08c4f4fd50d964688fb89"}
Sep 17 15:55:33.290883 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:33.290848    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:33.291192 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:33.291166    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:33.291303 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:33.291266    2585 scope.go:117] "RemoveContainer" containerID="1d093dfe283727be95adeabe3f981f92f6251f60be196ae08154fd1ac6b4fdb1"
Sep 17 15:55:33.291492 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:33.291470    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-l7rxh_openshift-console-operator(53c5829f-704f-4409-a7f6-95ac75a37a45)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" podUID="53c5829f-704f-4409-a7f6-95ac75a37a45"
Sep 17 15:55:33.905483 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:33.905432    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") pod \"image-registry-579d78b6fc-lbj92\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") " pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:55:33.905683 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:33.905616    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:33.905683 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:33.905638    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579d78b6fc-lbj92: secret "image-registry-tls" not found
Sep 17 15:55:33.905791 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:33.905717    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls podName:d416fdcc-1345-4037-b055-304ba1f0b3a5 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:37.905695945 +0000 UTC m=+164.171210939 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls") pod "image-registry-579d78b6fc-lbj92" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5") : secret "image-registry-tls" not found
Sep 17 15:55:34.005957 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:34.005915    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:55:34.006134 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:34.006044    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:55:34.006134 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:34.006117    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls podName:49b31cc2-74bc-48f2-8420-9bba8de27ab5 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:38.006102899 +0000 UTC m=+164.271617891 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls") pod "dns-default-dbb7d" (UID: "49b31cc2-74bc-48f2-8420-9bba8de27ab5") : secret "dns-default-metrics-tls" not found
Sep 17 15:55:34.255437 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:34.255407    2585 scope.go:117] "RemoveContainer" containerID="1d093dfe283727be95adeabe3f981f92f6251f60be196ae08154fd1ac6b4fdb1"
Sep 17 15:55:34.255891 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:34.255594    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-l7rxh_openshift-console-operator(53c5829f-704f-4409-a7f6-95ac75a37a45)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" podUID="53c5829f-704f-4409-a7f6-95ac75a37a45"
Sep 17 15:55:34.610483 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:34.610380    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:34.610661 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:34.610493    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:34.610661 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:34.610524    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:34.610661 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:34.610605    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:42.610584063 +0000 UTC m=+108.876099056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:34.610661 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:34.610617    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:34.610819 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:34.610669    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:42.610652127 +0000 UTC m=+108.876167125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:38.173064 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:38.173035    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-29lpf"
Sep 17 15:55:38.192026 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:38.191973    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-gz2xn" podStartSLOduration=6.391736118 podStartE2EDuration="8.191957958s" totalImagesPullingTime="1.80022184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:33.270783809 +0000 UTC m=+99.536298810" watchObservedRunningTime="2026-09-17 15:55:38.191957958 +0000 UTC m=+104.457473002"
Sep 17 15:55:38.842850 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:38.842799    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:38.843045 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:38.842964    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 15:55:38.843045 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:38.843035    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert podName:498bb02c-2602-4a5b-807d-bf5e8ad38c82 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:54.843019718 +0000 UTC m=+121.108534711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-dpcws" (UID: "498bb02c-2602-4a5b-807d-bf5e8ad38c82") : secret "networking-console-plugin-cert" not found
Sep 17 15:55:42.672785 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:42.672725    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:42.673311 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:42.672821    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:42.673311 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:42.672880    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:42.673311 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:42.672946    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls podName:3be4d3a6-64d1-4da1-a1e9-59fef88b7345 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:58.672931259 +0000 UTC m=+124.938446236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mrxm" (UID: "3be4d3a6-64d1-4da1-a1e9-59fef88b7345") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:42.673430 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:42.673340    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:45.779012 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:45.778980    2585 scope.go:117] "RemoveContainer" containerID="1d093dfe283727be95adeabe3f981f92f6251f60be196ae08154fd1ac6b4fdb1"
Sep 17 15:55:46.295044 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:46.295020    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 15:55:46.295437 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:46.295422    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/1.log"
Sep 17 15:55:46.295505 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:46.295453    2585 generic.go:356] "Generic (PLEG): container finished" podID="53c5829f-704f-4409-a7f6-95ac75a37a45" containerID="8734d9d4767a539b2a02faa852ff5aaab7b5ac3e103dd95755e722a9a04950f1" exitCode=255
Sep 17 15:55:46.295505 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:46.295498    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" event={"ID":"53c5829f-704f-4409-a7f6-95ac75a37a45","Type":"ContainerDied","Data":"8734d9d4767a539b2a02faa852ff5aaab7b5ac3e103dd95755e722a9a04950f1"}
Sep 17 15:55:46.295577 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:46.295543    2585 scope.go:117] "RemoveContainer" containerID="1d093dfe283727be95adeabe3f981f92f6251f60be196ae08154fd1ac6b4fdb1"
Sep 17 15:55:46.295897 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:46.295878    2585 scope.go:117] "RemoveContainer" containerID="8734d9d4767a539b2a02faa852ff5aaab7b5ac3e103dd95755e722a9a04950f1"
Sep 17 15:55:46.296063 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:46.296042    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-l7rxh_openshift-console-operator(53c5829f-704f-4409-a7f6-95ac75a37a45)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" podUID="53c5829f-704f-4409-a7f6-95ac75a37a45"
Sep 17 15:55:47.299880 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:47.299853    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 15:55:53.291200 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:53.291143    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:53.291655 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:53.291226    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:55:53.291655 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:53.291554    2585 scope.go:117] "RemoveContainer" containerID="8734d9d4767a539b2a02faa852ff5aaab7b5ac3e103dd95755e722a9a04950f1"
Sep 17 15:55:53.291746 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:55:53.291724    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-l7rxh_openshift-console-operator(53c5829f-704f-4409-a7f6-95ac75a37a45)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" podUID="53c5829f-704f-4409-a7f6-95ac75a37a45"
Sep 17 15:55:54.873632 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:54.873577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:54.876318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:54.876274    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/498bb02c-2602-4a5b-807d-bf5e8ad38c82-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-dpcws\" (UID: \"498bb02c-2602-4a5b-807d-bf5e8ad38c82\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:55.171794 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:55.171703    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"
Sep 17 15:55:55.449318 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:55.449078    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-dpcws"]
Sep 17 15:55:55.451839 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:55:55.451799    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498bb02c_2602_4a5b_807d_bf5e8ad38c82.slice/crio-2a67a977d4fbe03a205971ee3b9e753cfdb8d297ec3b64aa36eda45102ac2033 WatchSource:0}: Error finding container 2a67a977d4fbe03a205971ee3b9e753cfdb8d297ec3b64aa36eda45102ac2033: Status 404 returned error can't find the container with id 2a67a977d4fbe03a205971ee3b9e753cfdb8d297ec3b64aa36eda45102ac2033
Sep 17 15:55:56.329632 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:56.329587    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws" event={"ID":"498bb02c-2602-4a5b-807d-bf5e8ad38c82","Type":"ContainerStarted","Data":"2a67a977d4fbe03a205971ee3b9e753cfdb8d297ec3b64aa36eda45102ac2033"}
Sep 17 15:55:57.334460 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:57.334423    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws" event={"ID":"498bb02c-2602-4a5b-807d-bf5e8ad38c82","Type":"ContainerStarted","Data":"132941856aec46e17df09e2e8ddae9b4d29dacc6555e8082032d6bbb73335edb"}
Sep 17 15:55:58.704818 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:58.704779    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:58.707234 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:58.707213    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3be4d3a6-64d1-4da1-a1e9-59fef88b7345-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mrxm\" (UID: \"3be4d3a6-64d1-4da1-a1e9-59fef88b7345\") " pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:58.989396 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:58.989313    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9ppb\""
Sep 17 15:55:58.997382 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:58.997341    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5mrxm"
Sep 17 15:55:59.274818 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:59.274766    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-dpcws" podStartSLOduration=35.210294838 podStartE2EDuration="36.274746103s" totalImagesPullingTime="1.064451265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:57.360239325 +0000 UTC m=+123.625754324" watchObservedRunningTime="2026-09-17 15:55:59.274746103 +0000 UTC m=+125.540261102"
Sep 17 15:55:59.275636 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:59.275598    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5mrxm"]
Sep 17 15:55:59.278618 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:55:59.278593    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be4d3a6_64d1_4da1_a1e9_59fef88b7345.slice/crio-11b870d59d68ecc47208a259571ea65739d301dd1a219fe8b5cb5aef19f99c7a WatchSource:0}: Error finding container 11b870d59d68ecc47208a259571ea65739d301dd1a219fe8b5cb5aef19f99c7a: Status 404 returned error can't find the container with id 11b870d59d68ecc47208a259571ea65739d301dd1a219fe8b5cb5aef19f99c7a
Sep 17 15:55:59.342107 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:59.342077    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5mrxm" event={"ID":"3be4d3a6-64d1-4da1-a1e9-59fef88b7345","Type":"ContainerStarted","Data":"7b2deedae2b1afac78d849f1af0cc38a4f922286fe9a0a75d5ee881aed29706c"}
Sep 17 15:55:59.342217 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:55:59.342140    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5mrxm" event={"ID":"3be4d3a6-64d1-4da1-a1e9-59fef88b7345","Type":"ContainerStarted","Data":"11b870d59d68ecc47208a259571ea65739d301dd1a219fe8b5cb5aef19f99c7a"}
Sep 17 15:56:00.346965 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:00.346930    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5mrxm" event={"ID":"3be4d3a6-64d1-4da1-a1e9-59fef88b7345","Type":"ContainerStarted","Data":"a8fecebabe502340a57d36699214f249ab37130b63f2d659c4bc52f9fbd5b2e7"}
Sep 17 15:56:02.354807 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:02.354769    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5mrxm" event={"ID":"3be4d3a6-64d1-4da1-a1e9-59fef88b7345","Type":"ContainerStarted","Data":"61b7216a0e2e64b5f36c7fdd6b1f6e50ce646b51ed234acd435d5237b46edb5e"}
Sep 17 15:56:02.379810 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:02.379752    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5mrxm" podStartSLOduration=34.428027428 podStartE2EDuration="36.379731553s" totalImagesPullingTime="1.951704125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:02.378693962 +0000 UTC m=+128.644208961" watchObservedRunningTime="2026-09-17 15:56:02.379731553 +0000 UTC m=+128.645246553"
Sep 17 15:56:03.756259 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.756222    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"]
Sep 17 15:56:03.759429 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.759405    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"
Sep 17 15:56:03.762203 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.762183    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7pq4m\""
Sep 17 15:56:03.763358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.763339    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 15:56:03.763540 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.763518    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 15:56:03.763681 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.763662    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 15:56:03.776895 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.776867    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 15:56:03.781884 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.781858    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"]
Sep 17 15:56:03.883479 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.883443    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"]
Sep 17 15:56:03.887265 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.887248    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"
Sep 17 15:56:03.890581 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.890555    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 15:56:03.890701 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.890559    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x4t8w\""
Sep 17 15:56:03.911253 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.911227    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"]
Sep 17 15:56:03.946582 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.946547    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmlb\" (UniqueName: \"kubernetes.io/projected/71c172e7-d54a-4e91-bb04-c1d18ede24d8-kube-api-access-pgmlb\") pod \"managed-serviceaccount-addon-agent-676b7b95cb-qb9kx\" (UID: \"71c172e7-d54a-4e91-bb04-c1d18ede24d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"
Sep 17 15:56:03.946730 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:03.946605    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c172e7-d54a-4e91-bb04-c1d18ede24d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-676b7b95cb-qb9kx\" (UID: \"71c172e7-d54a-4e91-bb04-c1d18ede24d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"
Sep 17 15:56:04.047098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.047000    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmlb\" (UniqueName: \"kubernetes.io/projected/71c172e7-d54a-4e91-bb04-c1d18ede24d8-kube-api-access-pgmlb\") pod \"managed-serviceaccount-addon-agent-676b7b95cb-qb9kx\" (UID: \"71c172e7-d54a-4e91-bb04-c1d18ede24d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"
Sep 17 15:56:04.047098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.047075    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c172e7-d54a-4e91-bb04-c1d18ede24d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-676b7b95cb-qb9kx\" (UID: \"71c172e7-d54a-4e91-bb04-c1d18ede24d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"
Sep 17 15:56:04.047378 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.047110    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e3e53b0d-007e-44a2-af3a-35260192466d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8h4vk\" (UID: \"e3e53b0d-007e-44a2-af3a-35260192466d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"
Sep 17 15:56:04.050180 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.050153    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c172e7-d54a-4e91-bb04-c1d18ede24d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-676b7b95cb-qb9kx\" (UID: \"71c172e7-d54a-4e91-bb04-c1d18ede24d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"
Sep 17 15:56:04.077756 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.077729    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmlb\" (UniqueName: \"kubernetes.io/projected/71c172e7-d54a-4e91-bb04-c1d18ede24d8-kube-api-access-pgmlb\") pod \"managed-serviceaccount-addon-agent-676b7b95cb-qb9kx\" (UID: \"71c172e7-d54a-4e91-bb04-c1d18ede24d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"
Sep 17 15:56:04.078520 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.078499    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"
Sep 17 15:56:04.147779 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.147749    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e3e53b0d-007e-44a2-af3a-35260192466d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8h4vk\" (UID: \"e3e53b0d-007e-44a2-af3a-35260192466d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"
Sep 17 15:56:04.150940 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.150914    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e3e53b0d-007e-44a2-af3a-35260192466d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8h4vk\" (UID: \"e3e53b0d-007e-44a2-af3a-35260192466d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"
Sep 17 15:56:04.195799 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.195757    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"
Sep 17 15:56:04.429641 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.429552    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx"]
Sep 17 15:56:04.432254 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:04.432229    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71c172e7_d54a_4e91_bb04_c1d18ede24d8.slice/crio-e4829cb67ab1f2f29d5300626c74d90a7e398ef902b8d523f8bc84d30847d3d2 WatchSource:0}: Error finding container e4829cb67ab1f2f29d5300626c74d90a7e398ef902b8d523f8bc84d30847d3d2: Status 404 returned error can't find the container with id e4829cb67ab1f2f29d5300626c74d90a7e398ef902b8d523f8bc84d30847d3d2
Sep 17 15:56:04.508553 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:04.508520    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e53b0d_007e_44a2_af3a_35260192466d.slice/crio-60eb4bc2dd22e1649efcf58d60cd94cc6908389d8d9a26533125fc586f3bccdb WatchSource:0}: Error finding container 60eb4bc2dd22e1649efcf58d60cd94cc6908389d8d9a26533125fc586f3bccdb: Status 404 returned error can't find the container with id 60eb4bc2dd22e1649efcf58d60cd94cc6908389d8d9a26533125fc586f3bccdb
Sep 17 15:56:04.515184 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:04.515158    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"]
Sep 17 15:56:05.366528 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:05.366447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk" event={"ID":"e3e53b0d-007e-44a2-af3a-35260192466d","Type":"ContainerStarted","Data":"60eb4bc2dd22e1649efcf58d60cd94cc6908389d8d9a26533125fc586f3bccdb"}
Sep 17 15:56:05.368436 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:05.368382    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx" event={"ID":"71c172e7-d54a-4e91-bb04-c1d18ede24d8","Type":"ContainerStarted","Data":"e4829cb67ab1f2f29d5300626c74d90a7e398ef902b8d523f8bc84d30847d3d2"}
Sep 17 15:56:05.561527 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:05.561495    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:56:05.565108 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:05.565081    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc9f293-dabb-4bff-b819-41dc8625ce82-metrics-certs\") pod \"network-metrics-daemon-xh8xc\" (UID: \"afc9f293-dabb-4bff-b819-41dc8625ce82\") " pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:56:05.593274 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:05.593241    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9smtj\""
Sep 17 15:56:05.601581 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:05.601552    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xh8xc"
Sep 17 15:56:05.780933 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:05.778938    2585 scope.go:117] "RemoveContainer" containerID="8734d9d4767a539b2a02faa852ff5aaab7b5ac3e103dd95755e722a9a04950f1"
Sep 17 15:56:05.780933 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:05.779137    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-l7rxh_openshift-console-operator(53c5829f-704f-4409-a7f6-95ac75a37a45)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" podUID="53c5829f-704f-4409-a7f6-95ac75a37a45"
Sep 17 15:56:05.903980 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:05.903893    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xh8xc"]
Sep 17 15:56:05.915070 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:05.915041    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc9f293_dabb_4bff_b819_41dc8625ce82.slice/crio-a2101fb0879827ff9227f8543a736ec849dbf2f441ef76f10975b5a76da0235d WatchSource:0}: Error finding container a2101fb0879827ff9227f8543a736ec849dbf2f441ef76f10975b5a76da0235d: Status 404 returned error can't find the container with id a2101fb0879827ff9227f8543a736ec849dbf2f441ef76f10975b5a76da0235d
Sep 17 15:56:06.373367 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:06.373326    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xh8xc" event={"ID":"afc9f293-dabb-4bff-b819-41dc8625ce82","Type":"ContainerStarted","Data":"a2101fb0879827ff9227f8543a736ec849dbf2f441ef76f10975b5a76da0235d"}
Sep 17 15:56:06.375107 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:06.375078    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk" event={"ID":"e3e53b0d-007e-44a2-af3a-35260192466d","Type":"ContainerStarted","Data":"83e5916deec58681817521e28d485d2b6b8b2d02817d87b34b15b20a1cbd245c"}
Sep 17 15:56:06.375387 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:06.375327    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"
Sep 17 15:56:06.380837 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:06.380812    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk"
Sep 17 15:56:06.418926 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:06.418697    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8h4vk" podStartSLOduration=2.385708221 podStartE2EDuration="3.418674908s" totalImagesPullingTime="1.032966687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:06.418266437 +0000 UTC m=+132.683781436" watchObservedRunningTime="2026-09-17 15:56:06.418674908 +0000 UTC m=+132.684189908"
Sep 17 15:56:07.380336 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:07.380223    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx" event={"ID":"71c172e7-d54a-4e91-bb04-c1d18ede24d8","Type":"ContainerStarted","Data":"7145328762c2b1f6fb7a51ec6f1d077ef586cfa1b1cc8811c089d4839fed4685"}
Sep 17 15:56:07.399742 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:07.399685    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-676b7b95cb-qb9kx" podStartSLOduration=1.83047263 podStartE2EDuration="4.39966662s" totalImagesPullingTime="2.56919399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:07.398675886 +0000 UTC m=+133.664190887" watchObservedRunningTime="2026-09-17 15:56:07.39966662 +0000 UTC m=+133.665181620"
Sep 17 15:56:08.385526 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:08.385486    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xh8xc" event={"ID":"afc9f293-dabb-4bff-b819-41dc8625ce82","Type":"ContainerStarted","Data":"9ad181f1ec218286f973d2d5d04e3549068bea5ed331d3a1780ce59fbad389a3"}
Sep 17 15:56:08.385941 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:08.385541    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xh8xc" event={"ID":"afc9f293-dabb-4bff-b819-41dc8625ce82","Type":"ContainerStarted","Data":"7b5a639ff8f4008a97a14c09902f06eafb5f82d57cd6ee74367b7ecc8aa7889c"}
Sep 17 15:56:08.405362 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:08.405300    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xh8xc" podStartSLOduration=131.686487039 podStartE2EDuration="2m13.405268194s" totalImagesPullingTime="1.718781155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:08.402564524 +0000 UTC m=+134.668079523" watchObservedRunningTime="2026-09-17 15:56:08.405268194 +0000 UTC m=+134.670783194"
Sep 17 15:56:13.099731 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.099691    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"]
Sep 17 15:56:13.103506 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.103486    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.110158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.110124    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:56:13.110696 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.110675    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:56:13.115851 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.115832    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 15:56:13.117049 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.117034    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 15:56:13.121962 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.121887    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"]
Sep 17 15:56:13.122105 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.122064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhk44\" (UniqueName: \"kubernetes.io/projected/e579e186-6e4e-4617-b942-e7822402a738-kube-api-access-hhk44\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.122171 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.122128    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e579e186-6e4e-4617-b942-e7822402a738-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.122171 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.122160    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.122270 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.122217    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.124704 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.124679    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2zg2g\""
Sep 17 15:56:13.125016 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.124996    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:56:13.201636 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.201591    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xdqnb"]
Sep 17 15:56:13.204813 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.204789    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.210238 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.210213    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:56:13.210461 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.210445    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:56:13.210567 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.210551    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:56:13.210659 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.210572    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w9x66\""
Sep 17 15:56:13.223002 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.222974    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-sys\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhk44\" (UniqueName: \"kubernetes.io/projected/e579e186-6e4e-4617-b942-e7822402a738-kube-api-access-hhk44\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.223158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223102    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e579e186-6e4e-4617-b942-e7822402a738-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.223158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223132    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.223158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-tls\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223187    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-root\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223217    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-wtmp\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223242    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-textfile\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223398 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:13.223268    2585 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 15:56:13.223398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223277    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-accelerators-collector-config\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223327    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.223398 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:13.223364    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-tls podName:e579e186-6e4e-4617-b942-e7822402a738 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:13.723344999 +0000 UTC m=+139.988860002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-6xbpt" (UID: "e579e186-6e4e-4617-b942-e7822402a738") : secret "openshift-state-metrics-tls" not found
Sep 17 15:56:13.223763 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223403    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6j8w\" (UniqueName: \"kubernetes.io/projected/46b75e77-1806-45e8-bfd9-2708776f5f99-kube-api-access-j6j8w\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223763 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223431    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223763 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223459    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46b75e77-1806-45e8-bfd9-2708776f5f99-metrics-client-ca\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.223907 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.223867    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e579e186-6e4e-4617-b942-e7822402a738-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.225948 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.225925    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.252471 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.252438    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhk44\" (UniqueName: \"kubernetes.io/projected/e579e186-6e4e-4617-b942-e7822402a738-kube-api-access-hhk44\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.324888 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.324842    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6j8w\" (UniqueName: \"kubernetes.io/projected/46b75e77-1806-45e8-bfd9-2708776f5f99-kube-api-access-j6j8w\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325100 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.324904    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325100 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.324939    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46b75e77-1806-45e8-bfd9-2708776f5f99-metrics-client-ca\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325100 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.324974    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-sys\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325260 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325056    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-tls\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325260 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325099    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-sys\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325260 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325144    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-root\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325260 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325182    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-wtmp\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325260 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325208    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-textfile\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325260 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:13.325222    2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 15:56:13.325260 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-accelerators-collector-config\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325587 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:13.325279    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-tls podName:46b75e77-1806-45e8-bfd9-2708776f5f99 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:13.825259243 +0000 UTC m=+140.090774238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-tls") pod "node-exporter-xdqnb" (UID: "46b75e77-1806-45e8-bfd9-2708776f5f99") : secret "node-exporter-tls" not found
Sep 17 15:56:13.325587 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325310    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-root\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325587 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325456    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-wtmp\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325751 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325648    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46b75e77-1806-45e8-bfd9-2708776f5f99-metrics-client-ca\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325751 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325690    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-textfile\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.325827 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.325785    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-accelerators-collector-config\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.328008 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.327979    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.334516 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.334486    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6j8w\" (UniqueName: \"kubernetes.io/projected/46b75e77-1806-45e8-bfd9-2708776f5f99-kube-api-access-j6j8w\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.729176 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.729127    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.731764 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.731741    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e579e186-6e4e-4617-b942-e7822402a738-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-6xbpt\" (UID: \"e579e186-6e4e-4617-b942-e7822402a738\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:13.830649 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.830612    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-tls\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:13.833097 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:13.833058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46b75e77-1806-45e8-bfd9-2708776f5f99-node-exporter-tls\") pod \"node-exporter-xdqnb\" (UID: \"46b75e77-1806-45e8-bfd9-2708776f5f99\") " pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:14.014945 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.014909    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"
Sep 17 15:56:14.115337 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.115271    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xdqnb"
Sep 17 15:56:14.126054 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:14.126024    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b75e77_1806_45e8_bfd9_2708776f5f99.slice/crio-062df3a18c71990320889d9e8f749a17108c3ea4ac10e2f6d0aa9aa3738d1d6e WatchSource:0}: Error finding container 062df3a18c71990320889d9e8f749a17108c3ea4ac10e2f6d0aa9aa3738d1d6e: Status 404 returned error can't find the container with id 062df3a18c71990320889d9e8f749a17108c3ea4ac10e2f6d0aa9aa3738d1d6e
Sep 17 15:56:14.208023 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.207987    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:56:14.212693 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.212672    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.219097 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.219031    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:56:14.219097 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.219065    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:56:14.219097 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.219065    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:56:14.219401 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.219069    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:56:14.219794 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.219540    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-74t48\""
Sep 17 15:56:14.220312 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.220272    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:56:14.222736 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.222698    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:56:14.222736 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.222733    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:56:14.222957 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.222759    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:56:14.222957 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.222763    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:56:14.234704 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.234680    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.234831 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.234729    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.234831 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.234761    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.234831 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.234807    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-web-config\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.234988 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.234862    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.234988 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.234910    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.234988 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.234955    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.235138 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.235019    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.235138 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.235046    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.235138 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.235098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.235138 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.235128    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.235322 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.235153    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv5kh\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-kube-api-access-tv5kh\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.235322 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.235179    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-config-out\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.242021 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.241996    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:56:14.304813 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.304738    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt"]
Sep 17 15:56:14.307804 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:14.307774    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode579e186_6e4e_4617_b942_e7822402a738.slice/crio-8ab1e414628e36c09ab96d21ee740959fa89e68d6d8063e4bfb9ccc627309c60 WatchSource:0}: Error finding container 8ab1e414628e36c09ab96d21ee740959fa89e68d6d8063e4bfb9ccc627309c60: Status 404 returned error can't find the container with id 8ab1e414628e36c09ab96d21ee740959fa89e68d6d8063e4bfb9ccc627309c60
Sep 17 15:56:14.335847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.335815    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.335847 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.335850    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336041 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.335883    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336041 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.335903    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336041 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.335924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv5kh\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-kube-api-access-tv5kh\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336041 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.335941    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-config-out\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336041 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.335965    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336041 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:14.335994    2585 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 15:56:14.336358 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:14.336081    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls podName:58eab671-28ff-456c-831d-9ebc1ef959f5 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:14.836057222 +0000 UTC m=+141.101572219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5") : secret "alertmanager-main-tls" not found
Sep 17 15:56:14.336358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.336002    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.336216    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.336245    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-web-config\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.336253    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.336302    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.336335    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.336704 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.336372    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.337158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.337003    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.337158 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:14.337117    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle podName:58eab671-28ff-456c-831d-9ebc1ef959f5 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:14.837100403 +0000 UTC m=+141.102615396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5") : configmap references non-existent config key: ca-bundle.crt
Sep 17 15:56:14.339974 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.339495    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.339974 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.339584    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.339974 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.339606    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.339974 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.339609    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-config-out\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.339974 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.339901    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-web-config\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.339974 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.339948    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.340474 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.340450    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.341094 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.341073    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.349181 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.349159    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv5kh\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-kube-api-access-tv5kh\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.407776 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.407741    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt" event={"ID":"e579e186-6e4e-4617-b942-e7822402a738","Type":"ContainerStarted","Data":"27c2d805849a3b0e3a756f56173683035400298be47898f12570a8b46028e330"}
Sep 17 15:56:14.407881 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.407823    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt" event={"ID":"e579e186-6e4e-4617-b942-e7822402a738","Type":"ContainerStarted","Data":"8ab1e414628e36c09ab96d21ee740959fa89e68d6d8063e4bfb9ccc627309c60"}
Sep 17 15:56:14.409056 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.409030    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xdqnb" event={"ID":"46b75e77-1806-45e8-bfd9-2708776f5f99","Type":"ContainerStarted","Data":"062df3a18c71990320889d9e8f749a17108c3ea4ac10e2f6d0aa9aa3738d1d6e"}
Sep 17 15:56:14.842245 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.842204    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.842441 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.842269    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.843205 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.843182    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:14.845528 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:14.845487    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:15.122950 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:15.122863    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:56:15.414391 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:15.414274    2585 generic.go:356] "Generic (PLEG): container finished" podID="46b75e77-1806-45e8-bfd9-2708776f5f99" containerID="dc231c108b7cd5dba6655d8946d7760239900eb2c51158d489393496fe220238" exitCode=0
Sep 17 15:56:15.414391 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:15.414364    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xdqnb" event={"ID":"46b75e77-1806-45e8-bfd9-2708776f5f99","Type":"ContainerDied","Data":"dc231c108b7cd5dba6655d8946d7760239900eb2c51158d489393496fe220238"}
Sep 17 15:56:15.416480 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:15.416453    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt" event={"ID":"e579e186-6e4e-4617-b942-e7822402a738","Type":"ContainerStarted","Data":"3029df61b3de98325617127b65094fb3949082ea130d0de369a15d8e959b2dc6"}
Sep 17 15:56:15.448260 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:15.448230    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:56:15.448995 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:15.448966    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58eab671_28ff_456c_831d_9ebc1ef959f5.slice/crio-42aae4307d8572cd4caad0c5f1ffa810a4a80ed06d44ef7604c7326d3a569ef4 WatchSource:0}: Error finding container 42aae4307d8572cd4caad0c5f1ffa810a4a80ed06d44ef7604c7326d3a569ef4: Status 404 returned error can't find the container with id 42aae4307d8572cd4caad0c5f1ffa810a4a80ed06d44ef7604c7326d3a569ef4
Sep 17 15:56:16.422435 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:16.422390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerStarted","Data":"42aae4307d8572cd4caad0c5f1ffa810a4a80ed06d44ef7604c7326d3a569ef4"}
Sep 17 15:56:16.425253 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:16.425214    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xdqnb" event={"ID":"46b75e77-1806-45e8-bfd9-2708776f5f99","Type":"ContainerStarted","Data":"3a661e2fe51831118da4827b9ad799f25e7e751635e0952cadfa22d41201c92e"}
Sep 17 15:56:16.425431 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:16.425310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xdqnb" event={"ID":"46b75e77-1806-45e8-bfd9-2708776f5f99","Type":"ContainerStarted","Data":"a6f7913bf2fe6595b52dcaef546cfd9a32787ddc87c432af36afd78288561e92"}
Sep 17 15:56:16.459634 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:16.459578    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xdqnb" podStartSLOduration=2.326722193 podStartE2EDuration="3.459561843s" totalImagesPullingTime="1.06042046s" totalInitContainerRuntime="72.41919ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:16.456927669 +0000 UTC m=+142.722442672" watchObservedRunningTime="2026-09-17 15:56:16.459561843 +0000 UTC m=+142.725076839"
Sep 17 15:56:17.429899 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.429863    2585 generic.go:356] "Generic (PLEG): container finished" podID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerID="a03ec5c16b2dce77ef5e077e26716f84115e1209890bf22265005005dac23a17" exitCode=0
Sep 17 15:56:17.430337 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.429955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerDied","Data":"a03ec5c16b2dce77ef5e077e26716f84115e1209890bf22265005005dac23a17"}
Sep 17 15:56:17.633543 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.633507    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-59f5f9546d-7m2xx"]
Sep 17 15:56:17.636605 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.636589    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.639787 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.639758    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 15:56:17.639936 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.639903    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7foc6h4rc6jd1\""
Sep 17 15:56:17.640009 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.639958    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 15:56:17.640009 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.639990    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ht7tj\""
Sep 17 15:56:17.640171 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.640142    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:56:17.640340 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.640219    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 15:56:17.652037 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.652012    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59f5f9546d-7m2xx"]
Sep 17 15:56:17.668626 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.668592    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af656f33-b308-4cc7-bd5f-a689eb612866-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.668626 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.668626    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af656f33-b308-4cc7-bd5f-a689eb612866-metrics-server-audit-profiles\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.668849 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.668643    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-client-ca-bundle\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.668849 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.668712    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-secret-metrics-server-tls\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.668849 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.668747    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af656f33-b308-4cc7-bd5f-a689eb612866-audit-log\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.669000 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.668862    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92bnc\" (UniqueName: \"kubernetes.io/projected/af656f33-b308-4cc7-bd5f-a689eb612866-kube-api-access-92bnc\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.669000 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.668907    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-secret-metrics-server-client-certs\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770009 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.769977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bnc\" (UniqueName: \"kubernetes.io/projected/af656f33-b308-4cc7-bd5f-a689eb612866-kube-api-access-92bnc\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770174 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.770018    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-secret-metrics-server-client-certs\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770174 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.770067    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af656f33-b308-4cc7-bd5f-a689eb612866-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770174 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.770083    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af656f33-b308-4cc7-bd5f-a689eb612866-metrics-server-audit-profiles\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770174 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.770106    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-client-ca-bundle\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770447 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.770245    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-secret-metrics-server-tls\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770447 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.770315    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af656f33-b308-4cc7-bd5f-a689eb612866-audit-log\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770777 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.770745    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af656f33-b308-4cc7-bd5f-a689eb612866-audit-log\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.770951 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.770901    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af656f33-b308-4cc7-bd5f-a689eb612866-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.771212 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.771184    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af656f33-b308-4cc7-bd5f-a689eb612866-metrics-server-audit-profiles\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.772907 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.772877    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-secret-metrics-server-client-certs\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.773016 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.772906    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-secret-metrics-server-tls\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.773016 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.772955    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af656f33-b308-4cc7-bd5f-a689eb612866-client-ca-bundle\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.784431 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.784398    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92bnc\" (UniqueName: \"kubernetes.io/projected/af656f33-b308-4cc7-bd5f-a689eb612866-kube-api-access-92bnc\") pod \"metrics-server-59f5f9546d-7m2xx\" (UID: \"af656f33-b308-4cc7-bd5f-a689eb612866\") " pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.823197 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.823155    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"]
Sep 17 15:56:17.827402 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.827383    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:17.830053 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.830032    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vpxfl\""
Sep 17 15:56:17.830133 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.830059    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 15:56:17.839111 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.839088    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"]
Sep 17 15:56:17.871710 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.871676    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lc9x7\" (UID: \"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:17.871901 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.871746    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gfpj\" (UniqueName: \"kubernetes.io/projected/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-kube-api-access-8gfpj\") pod \"monitoring-plugin-5984459ff8-lc9x7\" (UID: \"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:17.945714 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.945669    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:17.972225 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.972198    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lc9x7\" (UID: \"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:17.972408 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.972257    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gfpj\" (UniqueName: \"kubernetes.io/projected/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-kube-api-access-8gfpj\") pod \"monitoring-plugin-5984459ff8-lc9x7\" (UID: \"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:17.972460 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:17.972403    2585 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 15:56:17.972496 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:17.972488    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-monitoring-plugin-cert podName:ea5f39cc-4284-4ac5-a72a-75ebdc2d6056 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:18.472466147 +0000 UTC m=+144.737981142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-lc9x7" (UID: "ea5f39cc-4284-4ac5-a72a-75ebdc2d6056") : secret "monitoring-plugin-cert" not found
Sep 17 15:56:17.982068 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:17.982037    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gfpj\" (UniqueName: \"kubernetes.io/projected/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-kube-api-access-8gfpj\") pod \"monitoring-plugin-5984459ff8-lc9x7\" (UID: \"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:18.234499 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:18.234403    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59f5f9546d-7m2xx"]
Sep 17 15:56:18.237783 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:18.237744    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf656f33_b308_4cc7_bd5f_a689eb612866.slice/crio-eefcbfc143d80a4235e3fb5d3f1dd351d7aecc42c6120a253660a4f7d7f839ba WatchSource:0}: Error finding container eefcbfc143d80a4235e3fb5d3f1dd351d7aecc42c6120a253660a4f7d7f839ba: Status 404 returned error can't find the container with id eefcbfc143d80a4235e3fb5d3f1dd351d7aecc42c6120a253660a4f7d7f839ba
Sep 17 15:56:18.436050 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:18.435993    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx" event={"ID":"af656f33-b308-4cc7-bd5f-a689eb612866","Type":"ContainerStarted","Data":"eefcbfc143d80a4235e3fb5d3f1dd351d7aecc42c6120a253660a4f7d7f839ba"}
Sep 17 15:56:18.477712 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:18.477675    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lc9x7\" (UID: \"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:18.480617 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:18.480587    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea5f39cc-4284-4ac5-a72a-75ebdc2d6056-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lc9x7\" (UID: \"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:18.736087 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:18.736045    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:19.319163 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.319065    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"]
Sep 17 15:56:19.444243 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.444203    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerStarted","Data":"a0b03f8c83bf638e51c678683bbc8af45721a4e33ac6ba7b39ccb00618a849c1"}
Sep 17 15:56:19.444698 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.444310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerStarted","Data":"1a29d338d1c26c82cee7de4af5c14b8f2a94822dcb7d909202443acfac59a26a"}
Sep 17 15:56:19.444698 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.444342    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerStarted","Data":"cb8bd66601a8ea8fbb1c711a48919772d8f5e76efec573b7ae26d93aefdbaaff"}
Sep 17 15:56:19.533549 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.533516    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:56:19.537449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.537423    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.546322 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.546193    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:56:19.546322 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.546244    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:56:19.546322 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.546193    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:56:19.546322 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.546199    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:56:19.546576 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.546250    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:56:19.546576 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.546201    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:56:19.547166 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.547144    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:56:19.547257 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.547184    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:56:19.547257 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.547198    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:56:19.556085 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.556053    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:56:19.556402 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.556382    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:56:19.556860 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.556838    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:56:19.562623 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.562598    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cj57h\""
Sep 17 15:56:19.562736 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.562605    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:56:19.562789 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.562748    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:56:19.570801 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.570774    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8git38i6usrj\""
Sep 17 15:56:19.587980 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.587944    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588154 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.587986    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588154 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588019    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588154 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588071    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588308 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588177    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588348 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588330    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588406 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588387    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588461 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588432    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twprk\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-kube-api-access-twprk\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588541 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588555    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588584    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588692 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588615    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588692 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588678    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-web-config\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588700    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config-out\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588725    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588745    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588761    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588957 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588775    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588957 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588824    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.588957 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.588859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.608162 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.608124    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:56:19.687151 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:19.687116    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5f39cc_4284_4ac5_a72a_75ebdc2d6056.slice/crio-9b2f35ef7b87edd7b2bc4a06875b676f8e17ebe1b3aec69fb1cc79340c061fa2 WatchSource:0}: Error finding container 9b2f35ef7b87edd7b2bc4a06875b676f8e17ebe1b3aec69fb1cc79340c061fa2: Status 404 returned error can't find the container with id 9b2f35ef7b87edd7b2bc4a06875b676f8e17ebe1b3aec69fb1cc79340c061fa2
Sep 17 15:56:19.689337 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689436 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689368    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689436 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689393    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twprk\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-kube-api-access-twprk\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689436 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689418    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689441    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689543    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-web-config\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config-out\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689597 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689592    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689650    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689672    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689706    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689733    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689771    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689829    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.689890 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689858    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.690358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.689899    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.690358 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.690226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.691176 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.690941    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.693404 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.693212    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.693955 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.693927    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.695639 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.694642    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.695639 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.694751    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.695800 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.695715    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.696525 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.696498    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.697236 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.697110    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.697856 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.697830    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.698578 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.698511    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.698811 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.698782    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.699301 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.699261    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.700397 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.700063    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config-out\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.700496 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.700381    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.700984 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.700965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.701923 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.701897    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-web-config\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.703667 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.703648    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.703861 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.703839    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.713454 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.713417    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twprk\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-kube-api-access-twprk\") pod \"prometheus-k8s-0\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:19.777812 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.777777    2585 scope.go:117] "RemoveContainer" containerID="8734d9d4767a539b2a02faa852ff5aaab7b5ac3e103dd95755e722a9a04950f1"
Sep 17 15:56:19.849016 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:19.848930    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:20.252433 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.252395    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:56:20.256753 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:20.256722    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82b60ee_e2ee_4318_8b06_27dcac407bd1.slice/crio-616e61061557b69bdcff89ba05a455970800f729f899d44579d1da21e445a3dc WatchSource:0}: Error finding container 616e61061557b69bdcff89ba05a455970800f729f899d44579d1da21e445a3dc: Status 404 returned error can't find the container with id 616e61061557b69bdcff89ba05a455970800f729f899d44579d1da21e445a3dc
Sep 17 15:56:20.451480 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.451442    2585 generic.go:356] "Generic (PLEG): container finished" podID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4" exitCode=0
Sep 17 15:56:20.452023 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.451533    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerDied","Data":"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"}
Sep 17 15:56:20.452023 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.451631    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerStarted","Data":"616e61061557b69bdcff89ba05a455970800f729f899d44579d1da21e445a3dc"}
Sep 17 15:56:20.453987 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.453935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7" event={"ID":"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056","Type":"ContainerStarted","Data":"9b2f35ef7b87edd7b2bc4a06875b676f8e17ebe1b3aec69fb1cc79340c061fa2"}
Sep 17 15:56:20.457668 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.457648    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 15:56:20.457769 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.457751    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" event={"ID":"53c5829f-704f-4409-a7f6-95ac75a37a45","Type":"ContainerStarted","Data":"4574a23637694cb1a100fa28c2e9e833ef235d272141d0ca09dd4f6f773f7a4f"}
Sep 17 15:56:20.458121 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.458040    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:56:20.461832 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.461775    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt" event={"ID":"e579e186-6e4e-4617-b942-e7822402a738","Type":"ContainerStarted","Data":"61a7a1eae713ccdcb2f0c86b4786d336c36fd5e8cfe90eb8e101e170eb4db137"}
Sep 17 15:56:20.463949 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.463894    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx" event={"ID":"af656f33-b308-4cc7-bd5f-a689eb612866","Type":"ContainerStarted","Data":"228fa20ad498f12013b5a9fde854a3b11f7810ae3a5cc1a41db705d9c10c57d2"}
Sep 17 15:56:20.467789 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.467665    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerStarted","Data":"e4b7d5e4149abce7704d24c5d5116413cbf612ecd57f413727a6510bd232b258"}
Sep 17 15:56:20.467789 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.467729    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerStarted","Data":"c7c58d26db9a2c3b6d92ac6d3faf0ecb5c820c8c72c3bc6355525caee5c291f1"}
Sep 17 15:56:20.526831 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.526777    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-6xbpt" podStartSLOduration=2.035467433 podStartE2EDuration="7.526761045s" totalImagesPullingTime="5.491293612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:20.526048291 +0000 UTC m=+146.791563293" watchObservedRunningTime="2026-09-17 15:56:20.526761045 +0000 UTC m=+146.792276095"
Sep 17 15:56:20.556802 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.556726    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx" podStartSLOduration=1.830644953 podStartE2EDuration="3.556701638s" totalImagesPullingTime="1.726056685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:20.555083685 +0000 UTC m=+146.820598697" watchObservedRunningTime="2026-09-17 15:56:20.556701638 +0000 UTC m=+146.822216639"
Sep 17 15:56:20.596396 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.594901    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh" podStartSLOduration=56.350776998 podStartE2EDuration="58.59487787s" totalImagesPullingTime="2.244100872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:20.593868321 +0000 UTC m=+146.859383325" watchObservedRunningTime="2026-09-17 15:56:20.59487787 +0000 UTC m=+146.860392873"
Sep 17 15:56:20.983186 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:20.983153    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-l7rxh"
Sep 17 15:56:21.474375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:21.474330    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7" event={"ID":"ea5f39cc-4284-4ac5-a72a-75ebdc2d6056","Type":"ContainerStarted","Data":"8c0f042b2df60f9a5ad4dca94b0a19c35f438737bcf51dcb17b0cfe179b528a1"}
Sep 17 15:56:21.474821 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:21.474569    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:21.479044 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:21.478609    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerStarted","Data":"896dd5a0db7f340bf9fe3c917cb92a84dda2ea55109f01ce21781df98e9f6bda"}
Sep 17 15:56:21.480489 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:21.480465    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7"
Sep 17 15:56:21.503962 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:21.503893    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lc9x7" podStartSLOduration=2.98772531 podStartE2EDuration="4.503871829s" totalImagesPullingTime="1.516146519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:21.502350994 +0000 UTC m=+147.767865997" watchObservedRunningTime="2026-09-17 15:56:21.503871829 +0000 UTC m=+147.769386830"
Sep 17 15:56:21.598327 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:21.598183    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.693126888 podStartE2EDuration="7.598158744s" totalImagesPullingTime="3.887914865s" totalInitContainerRuntime="17.116991ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:21.5948443 +0000 UTC m=+147.860359301" watchObservedRunningTime="2026-09-17 15:56:21.598158744 +0000 UTC m=+147.863673746"
Sep 17 15:56:24.495007 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:24.494959    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerStarted","Data":"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"}
Sep 17 15:56:24.495493 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:24.495075    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerStarted","Data":"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"}
Sep 17 15:56:26.184647 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.184544    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579d78b6fc-lbj92"]
Sep 17 15:56:26.185073 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:26.184879    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-579d78b6fc-lbj92" podUID="d416fdcc-1345-4037-b055-304ba1f0b3a5"
Sep 17 15:56:26.506491 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.506465    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:56:26.506636 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.506496    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerStarted","Data":"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"}
Sep 17 15:56:26.506636 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.506579    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerStarted","Data":"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"}
Sep 17 15:56:26.506636 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.506609    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerStarted","Data":"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"}
Sep 17 15:56:26.506754 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.506639    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerStarted","Data":"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"}
Sep 17 15:56:26.513149 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.513123    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:56:26.552738 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.552675    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.620081338 podStartE2EDuration="7.552645338s" totalImagesPullingTime="4.912164385s" totalInitContainerRuntime="20.399615ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:26.550740374 +0000 UTC m=+152.816255375" watchObservedRunningTime="2026-09-17 15:56:26.552645338 +0000 UTC m=+152.818160339"
Sep 17 15:56:26.660217 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660186    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d416fdcc-1345-4037-b055-304ba1f0b3a5-ca-trust-extracted\") pod \"d416fdcc-1345-4037-b055-304ba1f0b3a5\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") "
Sep 17 15:56:26.660380 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660231    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-bound-sa-token\") pod \"d416fdcc-1345-4037-b055-304ba1f0b3a5\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") "
Sep 17 15:56:26.660380 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660249    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngqmx\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-kube-api-access-ngqmx\") pod \"d416fdcc-1345-4037-b055-304ba1f0b3a5\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") "
Sep 17 15:56:26.660380 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660279    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-installation-pull-secrets\") pod \"d416fdcc-1345-4037-b055-304ba1f0b3a5\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") "
Sep 17 15:56:26.660522 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660398    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-certificates\") pod \"d416fdcc-1345-4037-b055-304ba1f0b3a5\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") "
Sep 17 15:56:26.660522 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660427    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-trusted-ca\") pod \"d416fdcc-1345-4037-b055-304ba1f0b3a5\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") "
Sep 17 15:56:26.660522 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660464    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-image-registry-private-configuration\") pod \"d416fdcc-1345-4037-b055-304ba1f0b3a5\" (UID: \"d416fdcc-1345-4037-b055-304ba1f0b3a5\") "
Sep 17 15:56:26.660522 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660483    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d416fdcc-1345-4037-b055-304ba1f0b3a5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d416fdcc-1345-4037-b055-304ba1f0b3a5" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:26.660768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660748    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d416fdcc-1345-4037-b055-304ba1f0b3a5-ca-trust-extracted\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:56:26.660821 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660756    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d416fdcc-1345-4037-b055-304ba1f0b3a5" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:26.660872 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.660823    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d416fdcc-1345-4037-b055-304ba1f0b3a5" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:26.662757 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.662731    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d416fdcc-1345-4037-b055-304ba1f0b3a5" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:26.662875 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.662775    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d416fdcc-1345-4037-b055-304ba1f0b3a5" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:26.662875 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.662806    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-kube-api-access-ngqmx" (OuterVolumeSpecName: "kube-api-access-ngqmx") pod "d416fdcc-1345-4037-b055-304ba1f0b3a5" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5"). InnerVolumeSpecName "kube-api-access-ngqmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:26.662875 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.662836    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d416fdcc-1345-4037-b055-304ba1f0b3a5" (UID: "d416fdcc-1345-4037-b055-304ba1f0b3a5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:26.761335 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.761232    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-certificates\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:56:26.761335 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.761272    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d416fdcc-1345-4037-b055-304ba1f0b3a5-trusted-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:56:26.761335 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.761295    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-image-registry-private-configuration\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:56:26.761335 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.761303    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-bound-sa-token\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:56:26.761335 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.761313    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqmx\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-kube-api-access-ngqmx\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:56:26.761335 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:26.761320    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d416fdcc-1345-4037-b055-304ba1f0b3a5-installation-pull-secrets\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:56:27.514379 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:27.511640    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579d78b6fc-lbj92"
Sep 17 15:56:27.551989 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:27.551958    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579d78b6fc-lbj92"]
Sep 17 15:56:27.557966 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:27.557942    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-579d78b6fc-lbj92"]
Sep 17 15:56:27.671173 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:27.671136    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d416fdcc-1345-4037-b055-304ba1f0b3a5-registry-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:56:27.782483 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:27.782395    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d416fdcc-1345-4037-b055-304ba1f0b3a5" path="/var/lib/kubelet/pods/d416fdcc-1345-4037-b055-304ba1f0b3a5/volumes"
Sep 17 15:56:29.849776 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:29.849743    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:56:32.533037 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:32.533004    2585 generic.go:356] "Generic (PLEG): container finished" podID="452444d7-ed37-4ddb-b83a-192ee0ef5727" containerID="9348b75a8e4d293596aa1431141f89d12f977be8cf06662799279cf09388ead2" exitCode=0
Sep 17 15:56:32.533420 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:32.533089    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f" event={"ID":"452444d7-ed37-4ddb-b83a-192ee0ef5727","Type":"ContainerDied","Data":"9348b75a8e4d293596aa1431141f89d12f977be8cf06662799279cf09388ead2"}
Sep 17 15:56:32.533509 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:32.533457    2585 scope.go:117] "RemoveContainer" containerID="9348b75a8e4d293596aa1431141f89d12f977be8cf06662799279cf09388ead2"
Sep 17 15:56:33.093356 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:56:33.093312    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dbb7d" podUID="49b31cc2-74bc-48f2-8420-9bba8de27ab5"
Sep 17 15:56:33.538788 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:33.538752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ts59f" event={"ID":"452444d7-ed37-4ddb-b83a-192ee0ef5727","Type":"ContainerStarted","Data":"84b333acd61329aedba7e3573ca96ae9318b196133a84dc53d12fba435825d53"}
Sep 17 15:56:33.538788 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:33.538781    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:56:36.999096 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:36.999016    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v5ms8_48fc9971-ac9e-4939-853c-cb4851a1f652/serve-healthcheck-canary/0.log"
Sep 17 15:56:37.946441 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:37.946387    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:37.946627 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:37.946475    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:38.060963 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:38.060923    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:56:38.063440 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:38.063418    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b31cc2-74bc-48f2-8420-9bba8de27ab5-metrics-tls\") pod \"dns-default-dbb7d\" (UID: \"49b31cc2-74bc-48f2-8420-9bba8de27ab5\") " pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:56:38.342465 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:38.342431    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcdnd\""
Sep 17 15:56:38.350537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:38.350520    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:56:38.624618 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:38.624595    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbb7d"]
Sep 17 15:56:38.627123 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:56:38.627096    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b31cc2_74bc_48f2_8420_9bba8de27ab5.slice/crio-84aece57daf20c8343d4bcc2b6d0710128166b87d9d502961f36bc195c559348 WatchSource:0}: Error finding container 84aece57daf20c8343d4bcc2b6d0710128166b87d9d502961f36bc195c559348: Status 404 returned error can't find the container with id 84aece57daf20c8343d4bcc2b6d0710128166b87d9d502961f36bc195c559348
Sep 17 15:56:39.565371 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:39.565330    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbb7d" event={"ID":"49b31cc2-74bc-48f2-8420-9bba8de27ab5","Type":"ContainerStarted","Data":"84aece57daf20c8343d4bcc2b6d0710128166b87d9d502961f36bc195c559348"}
Sep 17 15:56:41.575515 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:41.575480    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbb7d" event={"ID":"49b31cc2-74bc-48f2-8420-9bba8de27ab5","Type":"ContainerStarted","Data":"9fdd003677df0255e40a3570713d46f90cf47049508200060ce8062dcbbe5b14"}
Sep 17 15:56:41.575958 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:41.575570    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbb7d" event={"ID":"49b31cc2-74bc-48f2-8420-9bba8de27ab5","Type":"ContainerStarted","Data":"7050c3e036a0b793f3d501f021cde74a7a4f9f4a2a28eba307acf221d2966513"}
Sep 17 15:56:41.575958 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:41.575606    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:56:41.595828 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:41.595778    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dbb7d" podStartSLOduration=129.617679951 podStartE2EDuration="2m11.595763557s" totalImagesPullingTime="1.978083606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:41.594162441 +0000 UTC m=+167.859677442" watchObservedRunningTime="2026-09-17 15:56:41.595763557 +0000 UTC m=+167.861278557"
Sep 17 15:56:51.582055 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:51.582021    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbb7d"
Sep 17 15:56:57.950959 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:57.950929    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:56:57.954851 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:56:57.954829    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-59f5f9546d-7m2xx"
Sep 17 15:57:19.849888 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:19.849831    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:19.869226 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:19.869201    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:20.750200 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:20.750174    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:33.621591 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.621553    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:33.622127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.621841    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="alertmanager" containerID="cri-o://cb8bd66601a8ea8fbb1c711a48919772d8f5e76efec573b7ae26d93aefdbaaff" gracePeriod=120
Sep 17 15:57:33.622127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.621889    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy-web" containerID="cri-o://a0b03f8c83bf638e51c678683bbc8af45721a4e33ac6ba7b39ccb00618a849c1" gracePeriod=120
Sep 17 15:57:33.622127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.621927    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="config-reloader" containerID="cri-o://1a29d338d1c26c82cee7de4af5c14b8f2a94822dcb7d909202443acfac59a26a" gracePeriod=120
Sep 17 15:57:33.622127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.621869    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy-metric" containerID="cri-o://e4b7d5e4149abce7704d24c5d5116413cbf612ecd57f413727a6510bd232b258" gracePeriod=120
Sep 17 15:57:33.622127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.621953    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy" containerID="cri-o://c7c58d26db9a2c3b6d92ac6d3faf0ecb5c820c8c72c3bc6355525caee5c291f1" gracePeriod=120
Sep 17 15:57:33.622127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.621931    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="prom-label-proxy" containerID="cri-o://896dd5a0db7f340bf9fe3c917cb92a84dda2ea55109f01ce21781df98e9f6bda" gracePeriod=120
Sep 17 15:57:33.790903 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.790870    2585 generic.go:356] "Generic (PLEG): container finished" podID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerID="896dd5a0db7f340bf9fe3c917cb92a84dda2ea55109f01ce21781df98e9f6bda" exitCode=0
Sep 17 15:57:33.790903 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.790897    2585 generic.go:356] "Generic (PLEG): container finished" podID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerID="c7c58d26db9a2c3b6d92ac6d3faf0ecb5c820c8c72c3bc6355525caee5c291f1" exitCode=0
Sep 17 15:57:33.790903 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.790903    2585 generic.go:356] "Generic (PLEG): container finished" podID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerID="1a29d338d1c26c82cee7de4af5c14b8f2a94822dcb7d909202443acfac59a26a" exitCode=0
Sep 17 15:57:33.790903 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.790909    2585 generic.go:356] "Generic (PLEG): container finished" podID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerID="cb8bd66601a8ea8fbb1c711a48919772d8f5e76efec573b7ae26d93aefdbaaff" exitCode=0
Sep 17 15:57:33.791184 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.790944    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerDied","Data":"896dd5a0db7f340bf9fe3c917cb92a84dda2ea55109f01ce21781df98e9f6bda"}
Sep 17 15:57:33.791184 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.791009    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerDied","Data":"c7c58d26db9a2c3b6d92ac6d3faf0ecb5c820c8c72c3bc6355525caee5c291f1"}
Sep 17 15:57:33.791184 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.791035    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerDied","Data":"1a29d338d1c26c82cee7de4af5c14b8f2a94822dcb7d909202443acfac59a26a"}
Sep 17 15:57:33.791184 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:33.791058    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerDied","Data":"cb8bd66601a8ea8fbb1c711a48919772d8f5e76efec573b7ae26d93aefdbaaff"}
Sep 17 15:57:34.800424 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.800387    2585 generic.go:356] "Generic (PLEG): container finished" podID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerID="e4b7d5e4149abce7704d24c5d5116413cbf612ecd57f413727a6510bd232b258" exitCode=0
Sep 17 15:57:34.800424 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.800422    2585 generic.go:356] "Generic (PLEG): container finished" podID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerID="a0b03f8c83bf638e51c678683bbc8af45721a4e33ac6ba7b39ccb00618a849c1" exitCode=0
Sep 17 15:57:34.800942 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.800475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerDied","Data":"e4b7d5e4149abce7704d24c5d5116413cbf612ecd57f413727a6510bd232b258"}
Sep 17 15:57:34.800942 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.800551    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerDied","Data":"a0b03f8c83bf638e51c678683bbc8af45721a4e33ac6ba7b39ccb00618a849c1"}
Sep 17 15:57:34.888842 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.888819    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:34.975462 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975381    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975462 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975417    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975462 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975452    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-cluster-tls-config\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975735 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975482    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-main-db\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975735 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975510    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-web-config\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975735 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975535    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975735 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975584    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv5kh\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-kube-api-access-tv5kh\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975735 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975615    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-tls-assets\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975735 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975642    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975735 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975670    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-metrics-client-ca\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.975735 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975720    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-config-volume\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.976158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975762    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.976158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975785    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-config-out\") pod \"58eab671-28ff-456c-831d-9ebc1ef959f5\" (UID: \"58eab671-28ff-456c-831d-9ebc1ef959f5\") "
Sep 17 15:57:34.976158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.975814    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:34.976158 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.976034    2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-main-db\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:34.976741 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.976710    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:34.978559 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.978522    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:34.978879 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.978852    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-config-out" (OuterVolumeSpecName: "config-out") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:34.978879 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.978864    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:34.978989 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.978909    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:34.979130 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.979100    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:34.980050 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.980029    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:34.980364 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.980324    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:34.980866 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.980840    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-kube-api-access-tv5kh" (OuterVolumeSpecName: "kube-api-access-tv5kh") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "kube-api-access-tv5kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:34.981093 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.981072    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-config-volume" (OuterVolumeSpecName: "config-volume") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:34.985137 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.985108    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:34.991777 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:34.991754    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-web-config" (OuterVolumeSpecName: "web-config") pod "58eab671-28ff-456c-831d-9ebc1ef959f5" (UID: "58eab671-28ff-456c-831d-9ebc1ef959f5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:35.077369 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077337    2585 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-config-volume\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077369 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077367    2585 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077379    2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58eab671-28ff-456c-831d-9ebc1ef959f5-config-out\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077387    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-main-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077396    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077405    2585 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-cluster-tls-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077437    2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-web-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077446    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077456    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv5kh\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-kube-api-access-tv5kh\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077464    2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58eab671-28ff-456c-831d-9ebc1ef959f5-tls-assets\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077472    2585 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58eab671-28ff-456c-831d-9ebc1ef959f5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.077537 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.077480    2585 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58eab671-28ff-456c-831d-9ebc1ef959f5-metrics-client-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:35.807351 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.807320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58eab671-28ff-456c-831d-9ebc1ef959f5","Type":"ContainerDied","Data":"42aae4307d8572cd4caad0c5f1ffa810a4a80ed06d44ef7604c7326d3a569ef4"}
Sep 17 15:57:35.807719 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.807380    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.807719 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.807399    2585 scope.go:117] "RemoveContainer" containerID="896dd5a0db7f340bf9fe3c917cb92a84dda2ea55109f01ce21781df98e9f6bda"
Sep 17 15:57:35.815121 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.815103    2585 scope.go:117] "RemoveContainer" containerID="e4b7d5e4149abce7704d24c5d5116413cbf612ecd57f413727a6510bd232b258"
Sep 17 15:57:35.822734 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.822719    2585 scope.go:117] "RemoveContainer" containerID="c7c58d26db9a2c3b6d92ac6d3faf0ecb5c820c8c72c3bc6355525caee5c291f1"
Sep 17 15:57:35.829983 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.829957    2585 scope.go:117] "RemoveContainer" containerID="a0b03f8c83bf638e51c678683bbc8af45721a4e33ac6ba7b39ccb00618a849c1"
Sep 17 15:57:35.830070 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.830051    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:35.835747 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.835725    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:35.838460 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.838445    2585 scope.go:117] "RemoveContainer" containerID="1a29d338d1c26c82cee7de4af5c14b8f2a94822dcb7d909202443acfac59a26a"
Sep 17 15:57:35.845581 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.845565    2585 scope.go:117] "RemoveContainer" containerID="cb8bd66601a8ea8fbb1c711a48919772d8f5e76efec573b7ae26d93aefdbaaff"
Sep 17 15:57:35.852645 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.852626    2585 scope.go:117] "RemoveContainer" containerID="a03ec5c16b2dce77ef5e077e26716f84115e1209890bf22265005005dac23a17"
Sep 17 15:57:35.863902 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.863878    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:35.864149 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864138    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="config-reloader"
Sep 17 15:57:35.864193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864151    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="config-reloader"
Sep 17 15:57:35.864193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864163    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy"
Sep 17 15:57:35.864193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864170    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy"
Sep 17 15:57:35.864193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864178    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy-metric"
Sep 17 15:57:35.864193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864185    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy-metric"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864199    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="init-config-reloader"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864205    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="init-config-reloader"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864213    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="alertmanager"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864218    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="alertmanager"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864225    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy-web"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864230    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy-web"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864235    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="prom-label-proxy"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864240    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="prom-label-proxy"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864335    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="config-reloader"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864343    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy-metric"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864349    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy-web"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864356    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="kube-rbac-proxy"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864361    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="prom-label-proxy"
Sep 17 15:57:35.864398 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.864367    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" containerName="alertmanager"
Sep 17 15:57:35.869176 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.869159    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.871686 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.871660    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:57:35.871805 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.871714    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-74t48\""
Sep 17 15:57:35.871805 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.871665    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:57:35.871805 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.871666    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:57:35.872139 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.872120    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:57:35.872234 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.872194    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:57:35.872314 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.872257    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:57:35.872314 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.872307    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:57:35.872414 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.872398    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:57:35.877229 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.877209    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:57:35.884048 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.884024    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:35.983166 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983071    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983166 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983110    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdtl5\" (UniqueName: \"kubernetes.io/projected/778aa706-2049-4d00-b985-8aff84521b90-kube-api-access-zdtl5\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983166 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983161    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983497 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983195    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/778aa706-2049-4d00-b985-8aff84521b90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983497 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983213    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778aa706-2049-4d00-b985-8aff84521b90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983497 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983263    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778aa706-2049-4d00-b985-8aff84521b90-config-out\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983497 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983349    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983497 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983404    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778aa706-2049-4d00-b985-8aff84521b90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983497 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983432    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-web-config\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983497 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983465    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983497 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983490    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778aa706-2049-4d00-b985-8aff84521b90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983763 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983520    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-config-volume\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:35.983763 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:35.983570    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084476 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084433    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084626 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084489    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/778aa706-2049-4d00-b985-8aff84521b90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084626 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084598    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778aa706-2049-4d00-b985-8aff84521b90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084756 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084631    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778aa706-2049-4d00-b985-8aff84521b90-config-out\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084756 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084652    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084756 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084697    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778aa706-2049-4d00-b985-8aff84521b90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084756 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084716    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-web-config\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084756 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084739    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084994 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084767    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778aa706-2049-4d00-b985-8aff84521b90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084994 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084801    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-config-volume\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084994 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084852    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084994 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084893    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/778aa706-2049-4d00-b985-8aff84521b90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084994 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084897    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.084994 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.084925    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdtl5\" (UniqueName: \"kubernetes.io/projected/778aa706-2049-4d00-b985-8aff84521b90-kube-api-access-zdtl5\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.085886 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.085854    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778aa706-2049-4d00-b985-8aff84521b90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.088193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.087847    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778aa706-2049-4d00-b985-8aff84521b90-config-out\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.088193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.087899    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778aa706-2049-4d00-b985-8aff84521b90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.088193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.088129    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.088193 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.088149    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.088419 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.088242    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-config-volume\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.088419 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.088248    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-web-config\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.088579 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.088556    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778aa706-2049-4d00-b985-8aff84521b90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.088645 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.088626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.089091 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.089076    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.090385 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.090365    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778aa706-2049-4d00-b985-8aff84521b90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.093010 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.092987    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdtl5\" (UniqueName: \"kubernetes.io/projected/778aa706-2049-4d00-b985-8aff84521b90-kube-api-access-zdtl5\") pod \"alertmanager-main-0\" (UID: \"778aa706-2049-4d00-b985-8aff84521b90\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.178345 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.178304    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:57:36.490352 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.490241    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:57:36.493502 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:57:36.493475    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778aa706_2049_4d00_b985_8aff84521b90.slice/crio-b7c6801863934ab057bc37572db91d2b8ddb31a1c42c84911271aa5a8f7b0f30 WatchSource:0}: Error finding container b7c6801863934ab057bc37572db91d2b8ddb31a1c42c84911271aa5a8f7b0f30: Status 404 returned error can't find the container with id b7c6801863934ab057bc37572db91d2b8ddb31a1c42c84911271aa5a8f7b0f30
Sep 17 15:57:36.813998 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.813964    2585 generic.go:356] "Generic (PLEG): container finished" podID="778aa706-2049-4d00-b985-8aff84521b90" containerID="b2ecf16df106c0ee95a433cf62251a05f83a579b2146791505543f1781e12a80" exitCode=0
Sep 17 15:57:36.814389 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.814021    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778aa706-2049-4d00-b985-8aff84521b90","Type":"ContainerDied","Data":"b2ecf16df106c0ee95a433cf62251a05f83a579b2146791505543f1781e12a80"}
Sep 17 15:57:36.814389 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:36.814070    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778aa706-2049-4d00-b985-8aff84521b90","Type":"ContainerStarted","Data":"b7c6801863934ab057bc37572db91d2b8ddb31a1c42c84911271aa5a8f7b0f30"}
Sep 17 15:57:37.622882 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.622844    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cbbb68446-qgr42"]
Sep 17 15:57:37.626160 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.626145    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.630477 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.630445    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7gntp\""
Sep 17 15:57:37.630477 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.630468    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 15:57:37.630671 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.630487    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 15:57:37.630671 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.630631    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 15:57:37.630867 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.630848    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 15:57:37.630941 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.630894    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 15:57:37.635804 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.635783    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 15:57:37.643507 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.643484    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cbbb68446-qgr42"]
Sep 17 15:57:37.698059 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.698023    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.698194 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.698069    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-secret-telemeter-client\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.698194 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.698114    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-federate-client-tls\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.698194 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.698139    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.698339 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.698278    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2mwh\" (UniqueName: \"kubernetes.io/projected/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-kube-api-access-v2mwh\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.698399 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.698377    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-metrics-client-ca\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.698455 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.698409    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-telemeter-client-tls\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.698455 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.698444    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-serving-certs-ca-bundle\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.784660 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.784629    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58eab671-28ff-456c-831d-9ebc1ef959f5" path="/var/lib/kubelet/pods/58eab671-28ff-456c-831d-9ebc1ef959f5/volumes"
Sep 17 15:57:37.799825 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.799797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-metrics-client-ca\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.799918 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.799834    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-telemeter-client-tls\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.799918 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.799855    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-serving-certs-ca-bundle\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.799918 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.799899    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.800039 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.799929    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-secret-telemeter-client\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.800091 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.800057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-federate-client-tls\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.800144 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.800100    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.800249 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.800190    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mwh\" (UniqueName: \"kubernetes.io/projected/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-kube-api-access-v2mwh\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.800670 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.800643    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-serving-certs-ca-bundle\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.801019 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.800653    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-metrics-client-ca\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.801606 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.801584    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.802837 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.802629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-telemeter-client-tls\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.802837 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.802751    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.803109 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.803091    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-secret-telemeter-client\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.803146 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.803128    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-federate-client-tls\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.809729 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.809711    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2mwh\" (UniqueName: \"kubernetes.io/projected/e2d4036b-7351-48c3-9b8b-2e2cbf05e995-kube-api-access-v2mwh\") pod \"telemeter-client-6cbbb68446-qgr42\" (UID: \"e2d4036b-7351-48c3-9b8b-2e2cbf05e995\") " pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:37.822081 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.822058    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778aa706-2049-4d00-b985-8aff84521b90","Type":"ContainerStarted","Data":"45f73231b4fd48b9c9ca6015f1eb04018d478eb70e6b16b280972f95adc38354"}
Sep 17 15:57:37.822369 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.822116    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778aa706-2049-4d00-b985-8aff84521b90","Type":"ContainerStarted","Data":"a0dc06cc727ccf13029f1af3210138da48cee44c8d5acac8da35da174ac4feaf"}
Sep 17 15:57:37.822369 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.822141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778aa706-2049-4d00-b985-8aff84521b90","Type":"ContainerStarted","Data":"4feeabb63d21849e7bb208c99704bafbafc643cc121f659164a70ba29aaafb4f"}
Sep 17 15:57:37.822369 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.822166    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778aa706-2049-4d00-b985-8aff84521b90","Type":"ContainerStarted","Data":"ddf7e747a0db4e3ddd28ec6150d2a65dbbbfeb6996a2bffa4e81fb8fee5cc1da"}
Sep 17 15:57:37.822369 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.822190    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778aa706-2049-4d00-b985-8aff84521b90","Type":"ContainerStarted","Data":"6951c6d594ec34bb830fc324c6d4b9e0594f9cbd36857dd93579d77e37ead407"}
Sep 17 15:57:37.822369 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.822213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778aa706-2049-4d00-b985-8aff84521b90","Type":"ContainerStarted","Data":"1c9bf4631f662fd9a4059f6d0e7f192b32f21bb75aeb9279a32762979cf9529e"}
Sep 17 15:57:37.855084 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.855036    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.840443179 podStartE2EDuration="2.855019261s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.576082ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:37.853962695 +0000 UTC m=+224.119477696" watchObservedRunningTime="2026-09-17 15:57:37.855019261 +0000 UTC m=+224.120534262"
Sep 17 15:57:37.931831 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.931757    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:37.932059 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.932031    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="prometheus" containerID="cri-o://412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1" gracePeriod=600
Sep 17 15:57:37.932131 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.932046    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy" containerID="cri-o://13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c" gracePeriod=600
Sep 17 15:57:37.932131 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.932081    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="config-reloader" containerID="cri-o://fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf" gracePeriod=600
Sep 17 15:57:37.932131 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.932092    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="thanos-sidecar" containerID="cri-o://314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb" gracePeriod=600
Sep 17 15:57:37.932274 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.932143    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy-web" containerID="cri-o://a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988" gracePeriod=600
Sep 17 15:57:37.932274 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.932096    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742" gracePeriod=600
Sep 17 15:57:37.937252 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:37.937224    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42"
Sep 17 15:57:38.174407 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.174383    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:38.204592 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204529    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-metrics-client-certs\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204712 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204615    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-db\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204712 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204681    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-grpc-tls\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204833 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204714    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-web-config\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204833 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204755    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-kubelet-serving-ca-bundle\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204833 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204812    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204985 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204838    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-metrics-client-ca\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204985 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204865    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-trusted-ca-bundle\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204985 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204894    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-tls-assets\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204985 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204922    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-2\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204985 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204949    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-serving-certs-ca-bundle\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.204985 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.204983    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-thanos-prometheus-http-client-file\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205303 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205009    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-0\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205303 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205045    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-1\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205303 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205082    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twprk\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-kube-api-access-twprk\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205303 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205153    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:38.205729 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205323    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-kube-rbac-proxy\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205729 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205371    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config-out\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205729 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205412    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-tls\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205729 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205440    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205729 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205469    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config\") pod \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\" (UID: \"c82b60ee-e2ee-4318-8b06-27dcac407bd1\") "
Sep 17 15:57:38.205993 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205757    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.205993 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.205807    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:38.206216 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.206018    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:38.207336 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.206487    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:38.207336 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.206570    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:38.207961 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.207933    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.208315 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.208266    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:38.209104 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.208923    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:38.211676 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.211635    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:38.213671 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.213635    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.213810 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.213771    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.213931 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.213907    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.214029 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.214006    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.214084 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.214043    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config" (OuterVolumeSpecName: "config") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.214084 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.214061    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-kube-api-access-twprk" (OuterVolumeSpecName: "kube-api-access-twprk") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "kube-api-access-twprk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:38.214190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.214114    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:38.214190 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.214172    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config-out" (OuterVolumeSpecName: "config-out") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:38.214427 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.214406    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.216242 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.216215    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.224634 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.224612    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-web-config" (OuterVolumeSpecName: "web-config") pod "c82b60ee-e2ee-4318-8b06-27dcac407bd1" (UID: "c82b60ee-e2ee-4318-8b06-27dcac407bd1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:38.234661 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.234567    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cbbb68446-qgr42"]
Sep 17 15:57:38.238378 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:57:38.238357    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d4036b_7351_48c3_9b8b_2e2cbf05e995.slice/crio-76eb200a9d9302a0072c0447b54d8fc62ef65a3e2fd1f36df0383a5f46ae8788 WatchSource:0}: Error finding container 76eb200a9d9302a0072c0447b54d8fc62ef65a3e2fd1f36df0383a5f46ae8788: Status 404 returned error can't find the container with id 76eb200a9d9302a0072c0447b54d8fc62ef65a3e2fd1f36df0383a5f46ae8788
Sep 17 15:57:38.306225 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306204    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306225 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306223    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-metrics-client-ca\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306233    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306243    2585 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-tls-assets\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306252    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306260    2585 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306269    2585 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306278    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306308    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306318    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twprk\" (UniqueName: \"kubernetes.io/projected/c82b60ee-e2ee-4318-8b06-27dcac407bd1-kube-api-access-twprk\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306327    2585 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-kube-rbac-proxy\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306335    2585 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config-out\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306343    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306351    2585 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306359    2585 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306369    2585 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-metrics-client-certs\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306375 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306376    2585 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c82b60ee-e2ee-4318-8b06-27dcac407bd1-prometheus-k8s-db\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306823 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306384    2585 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-secret-grpc-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.306823 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.306411    2585 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c82b60ee-e2ee-4318-8b06-27dcac407bd1-web-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 15:57:38.827866 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.827832    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42" event={"ID":"e2d4036b-7351-48c3-9b8b-2e2cbf05e995","Type":"ContainerStarted","Data":"76eb200a9d9302a0072c0447b54d8fc62ef65a3e2fd1f36df0383a5f46ae8788"}
Sep 17 15:57:38.831738 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831709    2585 generic.go:356] "Generic (PLEG): container finished" podID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742" exitCode=0
Sep 17 15:57:38.831738 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831736    2585 generic.go:356] "Generic (PLEG): container finished" podID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c" exitCode=0
Sep 17 15:57:38.831921 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831745    2585 generic.go:356] "Generic (PLEG): container finished" podID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988" exitCode=0
Sep 17 15:57:38.831921 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831754    2585 generic.go:356] "Generic (PLEG): container finished" podID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb" exitCode=0
Sep 17 15:57:38.831921 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831762    2585 generic.go:356] "Generic (PLEG): container finished" podID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf" exitCode=0
Sep 17 15:57:38.831921 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831770    2585 generic.go:356] "Generic (PLEG): container finished" podID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1" exitCode=0
Sep 17 15:57:38.831921 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831795    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerDied","Data":"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"}
Sep 17 15:57:38.831921 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831814    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:38.831921 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831869    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerDied","Data":"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"}
Sep 17 15:57:38.831921 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831917    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerDied","Data":"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"}
Sep 17 15:57:38.832339 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831936    2585 scope.go:117] "RemoveContainer" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"
Sep 17 15:57:38.832339 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.831956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerDied","Data":"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"}
Sep 17 15:57:38.832339 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.832001    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerDied","Data":"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"}
Sep 17 15:57:38.832339 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.832044    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerDied","Data":"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"}
Sep 17 15:57:38.832339 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.832086    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c82b60ee-e2ee-4318-8b06-27dcac407bd1","Type":"ContainerDied","Data":"616e61061557b69bdcff89ba05a455970800f729f899d44579d1da21e445a3dc"}
Sep 17 15:57:38.841677 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.841656    2585 scope.go:117] "RemoveContainer" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"
Sep 17 15:57:38.851230 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.851201    2585 scope.go:117] "RemoveContainer" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"
Sep 17 15:57:38.858772 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.858749    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:38.861708 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.861688    2585 scope.go:117] "RemoveContainer" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"
Sep 17 15:57:38.862514 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.862492    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:38.869945 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.869925    2585 scope.go:117] "RemoveContainer" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"
Sep 17 15:57:38.879536 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.879206    2585 scope.go:117] "RemoveContainer" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"
Sep 17 15:57:38.888869 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.888847    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:38.889134 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889117    2585 scope.go:117] "RemoveContainer" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"
Sep 17 15:57:38.889256 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889241    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="prometheus"
Sep 17 15:57:38.889256 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889256    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="prometheus"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889268    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy-thanos"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889276    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy-thanos"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889361    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="config-reloader"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889373    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="config-reloader"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889386    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889395    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889407    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="init-config-reloader"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889413    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="init-config-reloader"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889425    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="thanos-sidecar"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889430    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="thanos-sidecar"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889439    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy-web"
Sep 17 15:57:38.889449 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889446    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy-web"
Sep 17 15:57:38.890037 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889534    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy"
Sep 17 15:57:38.890037 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889546    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy-thanos"
Sep 17 15:57:38.890037 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889556    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="prometheus"
Sep 17 15:57:38.890037 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889566    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="thanos-sidecar"
Sep 17 15:57:38.890037 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889578    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="config-reloader"
Sep 17 15:57:38.890037 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.889585    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" containerName="kube-rbac-proxy-web"
Sep 17 15:57:38.895105 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.895086    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:38.897745 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.897711    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:57:38.897846 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.897820    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:57:38.897975 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.897879    2585 scope.go:117] "RemoveContainer" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"
Sep 17 15:57:38.898229 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:57:38.898193    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": container with ID starting with 54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742 not found: ID does not exist" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"
Sep 17 15:57:38.898346 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898229    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:57:38.898346 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898232    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"} err="failed to get container status \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": rpc error: code = NotFound desc = could not find container \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": container with ID starting with 54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742 not found: ID does not exist"
Sep 17 15:57:38.898346 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898275    2585 scope.go:117] "RemoveContainer" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"
Sep 17 15:57:38.898346 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898306    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:57:38.898605 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898361    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:57:38.898605 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898471    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:57:38.898722 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898607    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:57:38.898722 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898668    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8git38i6usrj\""
Sep 17 15:57:38.898722 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:57:38.898677    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": container with ID starting with 13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c not found: ID does not exist" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"
Sep 17 15:57:38.898722 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898707    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"} err="failed to get container status \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": rpc error: code = NotFound desc = could not find container \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": container with ID starting with 13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c not found: ID does not exist"
Sep 17 15:57:38.898932 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898725    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:57:38.898932 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898730    2585 scope.go:117] "RemoveContainer" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"
Sep 17 15:57:38.898932 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898800    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:57:38.899077 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898963    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:57:38.899077 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898983    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cj57h\""
Sep 17 15:57:38.899077 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.898967    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:57:38.899077 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:57:38.899066    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": container with ID starting with a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988 not found: ID does not exist" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"
Sep 17 15:57:38.899303 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.899093    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"} err="failed to get container status \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": rpc error: code = NotFound desc = could not find container \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": container with ID starting with a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988 not found: ID does not exist"
Sep 17 15:57:38.899303 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.899111    2585 scope.go:117] "RemoveContainer" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"
Sep 17 15:57:38.899303 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.899201    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:57:38.899493 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:57:38.899428    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": container with ID starting with 314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb not found: ID does not exist" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"
Sep 17 15:57:38.899493 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.899456    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"} err="failed to get container status \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": rpc error: code = NotFound desc = could not find container \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": container with ID starting with 314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb not found: ID does not exist"
Sep 17 15:57:38.899493 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.899476    2585 scope.go:117] "RemoveContainer" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"
Sep 17 15:57:38.900048 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:57:38.900012    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": container with ID starting with fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf not found: ID does not exist" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"
Sep 17 15:57:38.900142 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.900058    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"} err="failed to get container status \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": rpc error: code = NotFound desc = could not find container \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": container with ID starting with fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf not found: ID does not exist"
Sep 17 15:57:38.900142 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.900086    2585 scope.go:117] "RemoveContainer" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"
Sep 17 15:57:38.900525 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:57:38.900498    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": container with ID starting with 412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1 not found: ID does not exist" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"
Sep 17 15:57:38.900630 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.900531    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"} err="failed to get container status \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": rpc error: code = NotFound desc = could not find container \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": container with ID starting with 412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1 not found: ID does not exist"
Sep 17 15:57:38.900630 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.900553    2585 scope.go:117] "RemoveContainer" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"
Sep 17 15:57:38.900929 ip-10-0-142-18 kubenswrapper[2585]: E0917 15:57:38.900830    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": container with ID starting with 5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4 not found: ID does not exist" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"
Sep 17 15:57:38.900929 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.900853    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"} err="failed to get container status \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": rpc error: code = NotFound desc = could not find container \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": container with ID starting with 5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4 not found: ID does not exist"
Sep 17 15:57:38.900929 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.900868    2585 scope.go:117] "RemoveContainer" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"
Sep 17 15:57:38.901228 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.901208    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:57:38.901228 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.901215    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"} err="failed to get container status \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": rpc error: code = NotFound desc = could not find container \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": container with ID starting with 54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742 not found: ID does not exist"
Sep 17 15:57:38.901426 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.901236    2585 scope.go:117] "RemoveContainer" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"
Sep 17 15:57:38.901626 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.901606    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"} err="failed to get container status \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": rpc error: code = NotFound desc = could not find container \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": container with ID starting with 13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c not found: ID does not exist"
Sep 17 15:57:38.901626 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.901626    2585 scope.go:117] "RemoveContainer" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"
Sep 17 15:57:38.901888 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.901869    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"} err="failed to get container status \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": rpc error: code = NotFound desc = could not find container \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": container with ID starting with a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988 not found: ID does not exist"
Sep 17 15:57:38.901963 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.901887    2585 scope.go:117] "RemoveContainer" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"
Sep 17 15:57:38.902268 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.902247    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"} err="failed to get container status \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": rpc error: code = NotFound desc = could not find container \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": container with ID starting with 314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb not found: ID does not exist"
Sep 17 15:57:38.902383 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.902270    2585 scope.go:117] "RemoveContainer" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"
Sep 17 15:57:38.902580 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.902561    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"} err="failed to get container status \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": rpc error: code = NotFound desc = could not find container \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": container with ID starting with fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf not found: ID does not exist"
Sep 17 15:57:38.902671 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.902582    2585 scope.go:117] "RemoveContainer" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"
Sep 17 15:57:38.902846 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.902829    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"} err="failed to get container status \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": rpc error: code = NotFound desc = could not find container \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": container with ID starting with 412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1 not found: ID does not exist"
Sep 17 15:57:38.902916 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.902847    2585 scope.go:117] "RemoveContainer" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"
Sep 17 15:57:38.903151 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.903124    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"} err="failed to get container status \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": rpc error: code = NotFound desc = could not find container \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": container with ID starting with 5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4 not found: ID does not exist"
Sep 17 15:57:38.903236 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.903153    2585 scope.go:117] "RemoveContainer" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"
Sep 17 15:57:38.903539 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.903450    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"} err="failed to get container status \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": rpc error: code = NotFound desc = could not find container \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": container with ID starting with 54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742 not found: ID does not exist"
Sep 17 15:57:38.903539 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.903476    2585 scope.go:117] "RemoveContainer" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"
Sep 17 15:57:38.903868 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.903843    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"} err="failed to get container status \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": rpc error: code = NotFound desc = could not find container \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": container with ID starting with 13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c not found: ID does not exist"
Sep 17 15:57:38.903937 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.903869    2585 scope.go:117] "RemoveContainer" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"
Sep 17 15:57:38.904171 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904151    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"} err="failed to get container status \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": rpc error: code = NotFound desc = could not find container \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": container with ID starting with a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988 not found: ID does not exist"
Sep 17 15:57:38.904263 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904172    2585 scope.go:117] "RemoveContainer" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"
Sep 17 15:57:38.904411 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904394    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:57:38.904484 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904440    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"} err="failed to get container status \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": rpc error: code = NotFound desc = could not find container \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": container with ID starting with 314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb not found: ID does not exist"
Sep 17 15:57:38.904484 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904458    2585 scope.go:117] "RemoveContainer" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"
Sep 17 15:57:38.904668 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904650    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"} err="failed to get container status \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": rpc error: code = NotFound desc = could not find container \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": container with ID starting with fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf not found: ID does not exist"
Sep 17 15:57:38.904724 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904668    2585 scope.go:117] "RemoveContainer" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"
Sep 17 15:57:38.904919 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904901    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"} err="failed to get container status \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": rpc error: code = NotFound desc = could not find container \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": container with ID starting with 412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1 not found: ID does not exist"
Sep 17 15:57:38.904987 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.904944    2585 scope.go:117] "RemoveContainer" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"
Sep 17 15:57:38.905187 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.905167    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"} err="failed to get container status \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": rpc error: code = NotFound desc = could not find container \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": container with ID starting with 5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4 not found: ID does not exist"
Sep 17 15:57:38.905187 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.905186    2585 scope.go:117] "RemoveContainer" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"
Sep 17 15:57:38.905500 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.905484    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"} err="failed to get container status \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": rpc error: code = NotFound desc = could not find container \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": container with ID starting with 54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742 not found: ID does not exist"
Sep 17 15:57:38.905602 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.905501    2585 scope.go:117] "RemoveContainer" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"
Sep 17 15:57:38.905723 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.905707    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"} err="failed to get container status \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": rpc error: code = NotFound desc = could not find container \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": container with ID starting with 13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c not found: ID does not exist"
Sep 17 15:57:38.905776 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.905725    2585 scope.go:117] "RemoveContainer" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"
Sep 17 15:57:38.905967 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.905945    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"} err="failed to get container status \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": rpc error: code = NotFound desc = could not find container \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": container with ID starting with a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988 not found: ID does not exist"
Sep 17 15:57:38.906008 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.905969    2585 scope.go:117] "RemoveContainer" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"
Sep 17 15:57:38.906215 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.906195    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"} err="failed to get container status \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": rpc error: code = NotFound desc = could not find container \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": container with ID starting with 314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb not found: ID does not exist"
Sep 17 15:57:38.906274 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.906215    2585 scope.go:117] "RemoveContainer" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"
Sep 17 15:57:38.906522 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.906505    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"} err="failed to get container status \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": rpc error: code = NotFound desc = could not find container \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": container with ID starting with fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf not found: ID does not exist"
Sep 17 15:57:38.906568 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.906523    2585 scope.go:117] "RemoveContainer" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"
Sep 17 15:57:38.906768 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.906749    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"} err="failed to get container status \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": rpc error: code = NotFound desc = could not find container \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": container with ID starting with 412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1 not found: ID does not exist"
Sep 17 15:57:38.906807 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.906769    2585 scope.go:117] "RemoveContainer" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"
Sep 17 15:57:38.907003 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.906987    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"} err="failed to get container status \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": rpc error: code = NotFound desc = could not find container \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": container with ID starting with 5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4 not found: ID does not exist"
Sep 17 15:57:38.907048 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907004    2585 scope.go:117] "RemoveContainer" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"
Sep 17 15:57:38.907219 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907202    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"} err="failed to get container status \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": rpc error: code = NotFound desc = could not find container \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": container with ID starting with 54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742 not found: ID does not exist"
Sep 17 15:57:38.907256 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907221    2585 scope.go:117] "RemoveContainer" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"
Sep 17 15:57:38.907478 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907461    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"} err="failed to get container status \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": rpc error: code = NotFound desc = could not find container \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": container with ID starting with 13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c not found: ID does not exist"
Sep 17 15:57:38.907523 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907477    2585 scope.go:117] "RemoveContainer" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"
Sep 17 15:57:38.907695 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907679    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"} err="failed to get container status \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": rpc error: code = NotFound desc = could not find container \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": container with ID starting with a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988 not found: ID does not exist"
Sep 17 15:57:38.907742 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907697    2585 scope.go:117] "RemoveContainer" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"
Sep 17 15:57:38.907919 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907903    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"} err="failed to get container status \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": rpc error: code = NotFound desc = could not find container \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": container with ID starting with 314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb not found: ID does not exist"
Sep 17 15:57:38.907919 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.907918    2585 scope.go:117] "RemoveContainer" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"
Sep 17 15:57:38.908130 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.908115    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"} err="failed to get container status \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": rpc error: code = NotFound desc = could not find container \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": container with ID starting with fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf not found: ID does not exist"
Sep 17 15:57:38.908172 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.908130    2585 scope.go:117] "RemoveContainer" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"
Sep 17 15:57:38.908386 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.908369    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"} err="failed to get container status \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": rpc error: code = NotFound desc = could not find container \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": container with ID starting with 412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1 not found: ID does not exist"
Sep 17 15:57:38.908436 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.908388    2585 scope.go:117] "RemoveContainer" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"
Sep 17 15:57:38.908617 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.908601    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"} err="failed to get container status \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": rpc error: code = NotFound desc = could not find container \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": container with ID starting with 5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4 not found: ID does not exist"
Sep 17 15:57:38.908659 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.908618    2585 scope.go:117] "RemoveContainer" containerID="54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"
Sep 17 15:57:38.908844 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.908829    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742"} err="failed to get container status \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": rpc error: code = NotFound desc = could not find container \"54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742\": container with ID starting with 54944e9b0837ef536fcf3056c6ee74d4d9534e71b3cd92ff1188d0a84bbee742 not found: ID does not exist"
Sep 17 15:57:38.908844 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.908844    2585 scope.go:117] "RemoveContainer" containerID="13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"
Sep 17 15:57:38.909038 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909022    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c"} err="failed to get container status \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": rpc error: code = NotFound desc = could not find container \"13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c\": container with ID starting with 13cf8092c662406e06fd4df0f7540ffbfc4d7bffec2f97415bbaeeb83b353e4c not found: ID does not exist"
Sep 17 15:57:38.909079 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909038    2585 scope.go:117] "RemoveContainer" containerID="a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"
Sep 17 15:57:38.909257 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909238    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988"} err="failed to get container status \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": rpc error: code = NotFound desc = could not find container \"a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988\": container with ID starting with a22bab1c792b71eb6dd4e624f41cc1d3fa739d08bf20685d69dd34967e1a0988 not found: ID does not exist"
Sep 17 15:57:38.909257 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909255    2585 scope.go:117] "RemoveContainer" containerID="314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"
Sep 17 15:57:38.909489 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909473    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb"} err="failed to get container status \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": rpc error: code = NotFound desc = could not find container \"314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb\": container with ID starting with 314c22bf9853a2bbd96c4555d81f58c29a3f95d576778aef365936bb8b85b7fb not found: ID does not exist"
Sep 17 15:57:38.909489 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909489    2585 scope.go:117] "RemoveContainer" containerID="fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"
Sep 17 15:57:38.909703 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909688    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf"} err="failed to get container status \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": rpc error: code = NotFound desc = could not find container \"fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf\": container with ID starting with fb33d6c076b906a8f0edb64e802ead9f29cee88240e3ab85190745a987543edf not found: ID does not exist"
Sep 17 15:57:38.909743 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909703    2585 scope.go:117] "RemoveContainer" containerID="412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"
Sep 17 15:57:38.909939 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909920    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1"} err="failed to get container status \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": rpc error: code = NotFound desc = could not find container \"412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1\": container with ID starting with 412bbf8cdc049402d513f58850e85d276e544e879e41c2dc25937912bd4f69b1 not found: ID does not exist"
Sep 17 15:57:38.909939 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.909938    2585 scope.go:117] "RemoveContainer" containerID="5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"
Sep 17 15:57:38.910275 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.910245    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4"} err="failed to get container status \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": rpc error: code = NotFound desc = could not find container \"5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4\": container with ID starting with 5d989e48da46e483aa506db8cf090fdabb11026b10427ab81493b63efd6037e4 not found: ID does not exist"
Sep 17 15:57:38.914553 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:38.914530    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:39.013033 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013008    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013196 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013039    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013196 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013058    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013196 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013075    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013196 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013196 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013146    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-config\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013196 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013183    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013251    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013365    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013407    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013437    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013460    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013492    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q866s\" (UniqueName: \"kubernetes.io/projected/492b25cd-faaf-489a-b68f-58369dca35a2-kube-api-access-q866s\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013573 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013560    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/492b25cd-faaf-489a-b68f-58369dca35a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013869 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013587    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/492b25cd-faaf-489a-b68f-58369dca35a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013869 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013612    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013869 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013641    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.013869 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.013658    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114097 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114144    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114173    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-config\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114197    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114238    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114279 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114310    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114343    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114372    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114447    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q866s\" (UniqueName: \"kubernetes.io/projected/492b25cd-faaf-489a-b68f-58369dca35a2-kube-api-access-q866s\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114519    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/492b25cd-faaf-489a-b68f-58369dca35a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114547    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/492b25cd-faaf-489a-b68f-58369dca35a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.114613 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114605    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.115098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114630    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.115098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114669    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.115098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.114702    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.115098 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.115078    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.115271 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.115244    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.116127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.115541    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.116127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.115575    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.116127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.115781    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.116127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.115827    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.116127 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.115880    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.117989 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.117694    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-config\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.119400 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.119043    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/492b25cd-faaf-489a-b68f-58369dca35a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.119400 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.119079    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.119400 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.119343    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.119849 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.119760    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/492b25cd-faaf-489a-b68f-58369dca35a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.120681 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.120656    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.121115 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.121061    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/492b25cd-faaf-489a-b68f-58369dca35a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.121115 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.121091    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.121975 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.121439    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.122136 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.122018    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.122803 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.122780    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.123911 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.123887    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/492b25cd-faaf-489a-b68f-58369dca35a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.125109 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.125089    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q866s\" (UniqueName: \"kubernetes.io/projected/492b25cd-faaf-489a-b68f-58369dca35a2-kube-api-access-q866s\") pod \"prometheus-k8s-0\" (UID: \"492b25cd-faaf-489a-b68f-58369dca35a2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.207121 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.207083    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:57:39.499529 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.499499    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:57:39.502675 ip-10-0-142-18 kubenswrapper[2585]: W0917 15:57:39.502642    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492b25cd_faaf_489a_b68f_58369dca35a2.slice/crio-2ad1bc7ea80b30668e06281b48e8424a182d7d4b587b6074c8b41fc4fe5250fb WatchSource:0}: Error finding container 2ad1bc7ea80b30668e06281b48e8424a182d7d4b587b6074c8b41fc4fe5250fb: Status 404 returned error can't find the container with id 2ad1bc7ea80b30668e06281b48e8424a182d7d4b587b6074c8b41fc4fe5250fb
Sep 17 15:57:39.788821 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.788746    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c82b60ee-e2ee-4318-8b06-27dcac407bd1" path="/var/lib/kubelet/pods/c82b60ee-e2ee-4318-8b06-27dcac407bd1/volumes"
Sep 17 15:57:39.840771 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.840730    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerStarted","Data":"e21cd8dd36de792b75006f10ab825bdf0dae372e08eda6b6b65f64c17b0d39ad"}
Sep 17 15:57:39.841231 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:39.840821    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerStarted","Data":"2ad1bc7ea80b30668e06281b48e8424a182d7d4b587b6074c8b41fc4fe5250fb"}
Sep 17 15:57:40.848394 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:40.848361    2585 generic.go:356] "Generic (PLEG): container finished" podID="492b25cd-faaf-489a-b68f-58369dca35a2" containerID="e21cd8dd36de792b75006f10ab825bdf0dae372e08eda6b6b65f64c17b0d39ad" exitCode=0
Sep 17 15:57:40.848833 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:40.848447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerDied","Data":"e21cd8dd36de792b75006f10ab825bdf0dae372e08eda6b6b65f64c17b0d39ad"}
Sep 17 15:57:40.850781 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:40.850760    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42" event={"ID":"e2d4036b-7351-48c3-9b8b-2e2cbf05e995","Type":"ContainerStarted","Data":"d5ac411e9c829ff350b592ad82b5544835523801864d80f3f07398325ee4b754"}
Sep 17 15:57:40.850866 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:40.850802    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42" event={"ID":"e2d4036b-7351-48c3-9b8b-2e2cbf05e995","Type":"ContainerStarted","Data":"a7cd572111840304714100706573be5856ecb2e40cdb2f7fbfd0557213b9daf9"}
Sep 17 15:57:40.850866 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:40.850820    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42" event={"ID":"e2d4036b-7351-48c3-9b8b-2e2cbf05e995","Type":"ContainerStarted","Data":"d827c25a88b362138f871c942ca127bbd64e00906a7ba696a40b898573c379b6"}
Sep 17 15:57:40.909361 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:40.909312    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cbbb68446-qgr42" podStartSLOduration=2.1183715850000002 podStartE2EDuration="3.909271798s" totalImagesPullingTime="1.790900213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:40.908761278 +0000 UTC m=+227.174276277" watchObservedRunningTime="2026-09-17 15:57:40.909271798 +0000 UTC m=+227.174786799"
Sep 17 15:57:41.859845 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:41.859806    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerStarted","Data":"dbaebad86a7de6f3ff0c4bbba785228d32509fbfcb886235dbdb9d43a7828de8"}
Sep 17 15:57:41.860353 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:41.859882    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerStarted","Data":"0d55473107da67dcb9cc1a6c6895f99f68bd3d3955eea436bb9efd39c76859e6"}
Sep 17 15:57:41.860353 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:41.859910    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerStarted","Data":"3912fc0de195cee6edee287ed38ccaa52414c2bd3ef99f3ddc2301511ca40d0d"}
Sep 17 15:57:41.860353 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:41.859935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerStarted","Data":"9ba1fd95644339f261dd78cf55a60ee010da7b8a4ea9de60cbd55e40deb78697"}
Sep 17 15:57:41.860353 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:41.859958    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerStarted","Data":"07cbe64aa8569fccd74afe6bf6d3631ed07b2543ae2056d17e02944b65a48548"}
Sep 17 15:57:41.860353 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:41.859987    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"492b25cd-faaf-489a-b68f-58369dca35a2","Type":"ContainerStarted","Data":"fe510bc9db380958b5f4db72be0033ffdb649777cb6a1b95c6b39e19666862dd"}
Sep 17 15:57:41.898033 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:41.897962    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.877692092 podStartE2EDuration="3.897938194s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.246102ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:41.895944293 +0000 UTC m=+228.161459293" watchObservedRunningTime="2026-09-17 15:57:41.897938194 +0000 UTC m=+228.163453197"
Sep 17 15:57:44.207809 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:57:44.207770    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:58:39.207361 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:58:39.207314    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:58:39.223766 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:58:39.223742    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:58:40.112737 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:58:40.112697    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:58:55.683368 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:58:55.683342    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 15:58:55.691095 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:58:55.691065    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 15:58:55.694545 ip-10-0-142-18 kubenswrapper[2585]: I0917 15:58:55.694525    2585 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:03:55.717560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:03:55.717532    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:03:55.723172 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:03:55.723148    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:04:26.018029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.017995    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-x6jp6"]
Sep 17 16:04:26.021239 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.021220    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6jp6"
Sep 17 16:04:26.025122 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.025091    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:04:26.025244 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.025125    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9gvx7\""
Sep 17 16:04:26.025737 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.025722    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 16:04:26.025783 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.025737    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:04:26.025817 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.025737    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:04:26.038979 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.038959    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x6jp6"]
Sep 17 16:04:26.132733 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.132696    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpnd\" (UniqueName: \"kubernetes.io/projected/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-kube-api-access-rcpnd\") pod \"s3-init-x6jp6\" (UID: \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\") " pod="kserve/s3-init-x6jp6"
Sep 17 16:04:26.132903 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.132752    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-model-cache\") pod \"s3-init-x6jp6\" (UID: \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\") " pod="kserve/s3-init-x6jp6"
Sep 17 16:04:26.233568 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.233518    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpnd\" (UniqueName: \"kubernetes.io/projected/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-kube-api-access-rcpnd\") pod \"s3-init-x6jp6\" (UID: \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\") " pod="kserve/s3-init-x6jp6"
Sep 17 16:04:26.233733 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.233597    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-model-cache\") pod \"s3-init-x6jp6\" (UID: \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\") " pod="kserve/s3-init-x6jp6"
Sep 17 16:04:26.233977 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.233958    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-model-cache\") pod \"s3-init-x6jp6\" (UID: \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\") " pod="kserve/s3-init-x6jp6"
Sep 17 16:04:26.243599 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.243568    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpnd\" (UniqueName: \"kubernetes.io/projected/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-kube-api-access-rcpnd\") pod \"s3-init-x6jp6\" (UID: \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\") " pod="kserve/s3-init-x6jp6"
Sep 17 16:04:26.331107 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.331022    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6jp6"
Sep 17 16:04:26.604122 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.604037    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x6jp6"]
Sep 17 16:04:26.607047 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:04:26.607020    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc77c66a1_2713_4ffc_8f9a_3ec101ba4315.slice/crio-b17cab009cfd1acc4c0e985f4fd72968f62accb12734b61f33b21f5569198e06 WatchSource:0}: Error finding container b17cab009cfd1acc4c0e985f4fd72968f62accb12734b61f33b21f5569198e06: Status 404 returned error can't find the container with id b17cab009cfd1acc4c0e985f4fd72968f62accb12734b61f33b21f5569198e06
Sep 17 16:04:26.609116 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:26.609099    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:04:27.514943 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:27.514896    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6jp6" event={"ID":"c77c66a1-2713-4ffc-8f9a-3ec101ba4315","Type":"ContainerStarted","Data":"b17cab009cfd1acc4c0e985f4fd72968f62accb12734b61f33b21f5569198e06"}
Sep 17 16:04:31.536440 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:31.536402    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6jp6" event={"ID":"c77c66a1-2713-4ffc-8f9a-3ec101ba4315","Type":"ContainerStarted","Data":"094f0cd688b9feb85baec08c07527d4b095d7dae6be8d7d9db68ae7903691a50"}
Sep 17 16:04:33.546198 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:33.546165    2585 generic.go:356] "Generic (PLEG): container finished" podID="c77c66a1-2713-4ffc-8f9a-3ec101ba4315" containerID="094f0cd688b9feb85baec08c07527d4b095d7dae6be8d7d9db68ae7903691a50" exitCode=0
Sep 17 16:04:33.546569 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:33.546236    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6jp6" event={"ID":"c77c66a1-2713-4ffc-8f9a-3ec101ba4315","Type":"ContainerDied","Data":"094f0cd688b9feb85baec08c07527d4b095d7dae6be8d7d9db68ae7903691a50"}
Sep 17 16:04:38.574839 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:38.574802    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6jp6" event={"ID":"c77c66a1-2713-4ffc-8f9a-3ec101ba4315","Type":"ContainerStarted","Data":"9eeac7679aa794bd624429e4890e904e6a55277a3912743abe7ffd157dc220da"}
Sep 17 16:04:38.592193 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:04:38.592108    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-x6jp6" podStartSLOduration=2.893643217 podStartE2EDuration="13.59209013s" totalImagesPullingTime="8.934881944s" totalInitContainerRuntime="1.763564969s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:04:38.591260175 +0000 UTC m=+644.856775176" watchObservedRunningTime="2026-09-17 16:04:38.59209013 +0000 UTC m=+644.857605129"
Sep 17 16:05:43.846175 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:43.846135    2585 generic.go:356] "Generic (PLEG): container finished" podID="c77c66a1-2713-4ffc-8f9a-3ec101ba4315" containerID="9eeac7679aa794bd624429e4890e904e6a55277a3912743abe7ffd157dc220da" exitCode=0
Sep 17 16:05:43.846634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:43.846209    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6jp6" event={"ID":"c77c66a1-2713-4ffc-8f9a-3ec101ba4315","Type":"ContainerDied","Data":"9eeac7679aa794bd624429e4890e904e6a55277a3912743abe7ffd157dc220da"}
Sep 17 16:05:44.974005 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:44.973979    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6jp6"
Sep 17 16:05:45.136626 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.136518    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-model-cache\") pod \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\" (UID: \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\") "
Sep 17 16:05:45.136783 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.136636    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcpnd\" (UniqueName: \"kubernetes.io/projected/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-kube-api-access-rcpnd\") pod \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\" (UID: \"c77c66a1-2713-4ffc-8f9a-3ec101ba4315\") "
Sep 17 16:05:45.138984 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.138953    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-kube-api-access-rcpnd" (OuterVolumeSpecName: "kube-api-access-rcpnd") pod "c77c66a1-2713-4ffc-8f9a-3ec101ba4315" (UID: "c77c66a1-2713-4ffc-8f9a-3ec101ba4315"). InnerVolumeSpecName "kube-api-access-rcpnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:05:45.156342 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.156299    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-model-cache" (OuterVolumeSpecName: "model-cache") pod "c77c66a1-2713-4ffc-8f9a-3ec101ba4315" (UID: "c77c66a1-2713-4ffc-8f9a-3ec101ba4315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:05:45.237475 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.237435    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcpnd\" (UniqueName: \"kubernetes.io/projected/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-kube-api-access-rcpnd\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:05:45.237475 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.237470    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77c66a1-2713-4ffc-8f9a-3ec101ba4315-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:05:45.855613 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.855583    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6jp6"
Sep 17 16:05:45.855779 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.855618    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6jp6" event={"ID":"c77c66a1-2713-4ffc-8f9a-3ec101ba4315","Type":"ContainerDied","Data":"b17cab009cfd1acc4c0e985f4fd72968f62accb12734b61f33b21f5569198e06"}
Sep 17 16:05:45.855779 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:45.855652    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17cab009cfd1acc4c0e985f4fd72968f62accb12734b61f33b21f5569198e06"
Sep 17 16:05:53.930471 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.930438    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-cvxvk"]
Sep 17 16:05:53.930845 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.930727    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77c66a1-2713-4ffc-8f9a-3ec101ba4315" containerName="download-hf-model"
Sep 17 16:05:53.930845 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.930738    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77c66a1-2713-4ffc-8f9a-3ec101ba4315" containerName="download-hf-model"
Sep 17 16:05:53.930845 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.930755    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77c66a1-2713-4ffc-8f9a-3ec101ba4315" containerName="s3-init"
Sep 17 16:05:53.930845 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.930760    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77c66a1-2713-4ffc-8f9a-3ec101ba4315" containerName="s3-init"
Sep 17 16:05:53.930845 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.930833    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77c66a1-2713-4ffc-8f9a-3ec101ba4315" containerName="s3-init"
Sep 17 16:05:53.933628 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.933612    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:05:53.936109 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.936072    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 16:05:53.936230 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.936133    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9gvx7\""
Sep 17 16:05:53.936230 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.936073    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:05:53.937014 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.936997    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:05:53.937127 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.937001    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:05:53.941366 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:53.941338    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-cvxvk"]
Sep 17 16:05:54.007067 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.007045    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt6rh\" (UniqueName: \"kubernetes.io/projected/8805fe9f-c9c5-447a-bad2-fabad597ad81-kube-api-access-wt6rh\") pod \"s3-tls-init-custom-cvxvk\" (UID: \"8805fe9f-c9c5-447a-bad2-fabad597ad81\") " pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:05:54.007213 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.007110    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8805fe9f-c9c5-447a-bad2-fabad597ad81-model-cache\") pod \"s3-tls-init-custom-cvxvk\" (UID: \"8805fe9f-c9c5-447a-bad2-fabad597ad81\") " pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:05:54.108254 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.108223    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8805fe9f-c9c5-447a-bad2-fabad597ad81-model-cache\") pod \"s3-tls-init-custom-cvxvk\" (UID: \"8805fe9f-c9c5-447a-bad2-fabad597ad81\") " pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:05:54.108419 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.108324    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt6rh\" (UniqueName: \"kubernetes.io/projected/8805fe9f-c9c5-447a-bad2-fabad597ad81-kube-api-access-wt6rh\") pod \"s3-tls-init-custom-cvxvk\" (UID: \"8805fe9f-c9c5-447a-bad2-fabad597ad81\") " pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:05:54.108608 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.108591    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8805fe9f-c9c5-447a-bad2-fabad597ad81-model-cache\") pod \"s3-tls-init-custom-cvxvk\" (UID: \"8805fe9f-c9c5-447a-bad2-fabad597ad81\") " pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:05:54.117277 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.117256    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt6rh\" (UniqueName: \"kubernetes.io/projected/8805fe9f-c9c5-447a-bad2-fabad597ad81-kube-api-access-wt6rh\") pod \"s3-tls-init-custom-cvxvk\" (UID: \"8805fe9f-c9c5-447a-bad2-fabad597ad81\") " pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:05:54.243394 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.243329    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:05:54.517114 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.517087    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-cvxvk"]
Sep 17 16:05:54.519650 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:05:54.519623    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8805fe9f_c9c5_447a_bad2_fabad597ad81.slice/crio-b0f164e2a491188b6e35e2aeceb0f48a116f5779714f094089b315e54d4415e6 WatchSource:0}: Error finding container b0f164e2a491188b6e35e2aeceb0f48a116f5779714f094089b315e54d4415e6: Status 404 returned error can't find the container with id b0f164e2a491188b6e35e2aeceb0f48a116f5779714f094089b315e54d4415e6
Sep 17 16:05:54.895094 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.894998    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cvxvk" event={"ID":"8805fe9f-c9c5-447a-bad2-fabad597ad81","Type":"ContainerStarted","Data":"e9b41d469d70d3e6b9e9b501115daedc725e84706d184632e4f970b53d0f2810"}
Sep 17 16:05:54.895094 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:54.895042    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cvxvk" event={"ID":"8805fe9f-c9c5-447a-bad2-fabad597ad81","Type":"ContainerStarted","Data":"b0f164e2a491188b6e35e2aeceb0f48a116f5779714f094089b315e54d4415e6"}
Sep 17 16:05:56.905941 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:56.905906    2585 generic.go:356] "Generic (PLEG): container finished" podID="8805fe9f-c9c5-447a-bad2-fabad597ad81" containerID="e9b41d469d70d3e6b9e9b501115daedc725e84706d184632e4f970b53d0f2810" exitCode=0
Sep 17 16:05:56.906336 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:56.905986    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cvxvk" event={"ID":"8805fe9f-c9c5-447a-bad2-fabad597ad81","Type":"ContainerDied","Data":"e9b41d469d70d3e6b9e9b501115daedc725e84706d184632e4f970b53d0f2810"}
Sep 17 16:05:56.909403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:56.909384    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 16:05:57.912360 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:05:57.912323    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cvxvk" event={"ID":"8805fe9f-c9c5-447a-bad2-fabad597ad81","Type":"ContainerStarted","Data":"14136ffe0fb5a04dd39ed17a4b0e3a67facae3d026df5999fed4c578c36c6de5"}
Sep 17 16:06:04.947462 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:04.947428    2585 generic.go:356] "Generic (PLEG): container finished" podID="8805fe9f-c9c5-447a-bad2-fabad597ad81" containerID="14136ffe0fb5a04dd39ed17a4b0e3a67facae3d026df5999fed4c578c36c6de5" exitCode=0
Sep 17 16:06:04.947885 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:04.947509    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cvxvk" event={"ID":"8805fe9f-c9c5-447a-bad2-fabad597ad81","Type":"ContainerDied","Data":"14136ffe0fb5a04dd39ed17a4b0e3a67facae3d026df5999fed4c578c36c6de5"}
Sep 17 16:06:06.096478 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.096451    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:06:06.218647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.218544    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt6rh\" (UniqueName: \"kubernetes.io/projected/8805fe9f-c9c5-447a-bad2-fabad597ad81-kube-api-access-wt6rh\") pod \"8805fe9f-c9c5-447a-bad2-fabad597ad81\" (UID: \"8805fe9f-c9c5-447a-bad2-fabad597ad81\") "
Sep 17 16:06:06.218647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.218649    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8805fe9f-c9c5-447a-bad2-fabad597ad81-model-cache\") pod \"8805fe9f-c9c5-447a-bad2-fabad597ad81\" (UID: \"8805fe9f-c9c5-447a-bad2-fabad597ad81\") "
Sep 17 16:06:06.220933 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.220893    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8805fe9f-c9c5-447a-bad2-fabad597ad81-kube-api-access-wt6rh" (OuterVolumeSpecName: "kube-api-access-wt6rh") pod "8805fe9f-c9c5-447a-bad2-fabad597ad81" (UID: "8805fe9f-c9c5-447a-bad2-fabad597ad81"). InnerVolumeSpecName "kube-api-access-wt6rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:06:06.268385 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.268343    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8805fe9f-c9c5-447a-bad2-fabad597ad81-model-cache" (OuterVolumeSpecName: "model-cache") pod "8805fe9f-c9c5-447a-bad2-fabad597ad81" (UID: "8805fe9f-c9c5-447a-bad2-fabad597ad81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:06:06.319463 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.319421    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt6rh\" (UniqueName: \"kubernetes.io/projected/8805fe9f-c9c5-447a-bad2-fabad597ad81-kube-api-access-wt6rh\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:06.319463 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.319453    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8805fe9f-c9c5-447a-bad2-fabad597ad81-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:06.957207 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.957181    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cvxvk"
Sep 17 16:06:06.957401 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.957200    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cvxvk" event={"ID":"8805fe9f-c9c5-447a-bad2-fabad597ad81","Type":"ContainerDied","Data":"b0f164e2a491188b6e35e2aeceb0f48a116f5779714f094089b315e54d4415e6"}
Sep 17 16:06:06.957401 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:06.957240    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0f164e2a491188b6e35e2aeceb0f48a116f5779714f094089b315e54d4415e6"
Sep 17 16:06:12.760316 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.760259    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-h9vjj"]
Sep 17 16:06:12.760734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.760630    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8805fe9f-c9c5-447a-bad2-fabad597ad81" containerName="download-hf-model"
Sep 17 16:06:12.760734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.760644    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8805fe9f-c9c5-447a-bad2-fabad597ad81" containerName="download-hf-model"
Sep 17 16:06:12.760734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.760671    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8805fe9f-c9c5-447a-bad2-fabad597ad81" containerName="s3-tls-init-custom"
Sep 17 16:06:12.760734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.760677    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8805fe9f-c9c5-447a-bad2-fabad597ad81" containerName="s3-tls-init-custom"
Sep 17 16:06:12.760904 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.760735    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8805fe9f-c9c5-447a-bad2-fabad597ad81" containerName="s3-tls-init-custom"
Sep 17 16:06:12.763906 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.763884    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:12.766442 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.766413    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:06:12.766442 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.766433    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:06:12.766637 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.766459    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 16:06:12.766637 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.766482    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:06:12.767539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.767521    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9gvx7\""
Sep 17 16:06:12.772096 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.772072    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-h9vjj"]
Sep 17 16:06:12.874247 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.874198    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v78lf\" (UniqueName: \"kubernetes.io/projected/6f75400d-a054-447b-a0ed-b38e302afe1d-kube-api-access-v78lf\") pod \"s3-tls-init-serving-h9vjj\" (UID: \"6f75400d-a054-447b-a0ed-b38e302afe1d\") " pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:12.874454 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.874267    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f75400d-a054-447b-a0ed-b38e302afe1d-model-cache\") pod \"s3-tls-init-serving-h9vjj\" (UID: \"6f75400d-a054-447b-a0ed-b38e302afe1d\") " pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:12.975716 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.975664    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f75400d-a054-447b-a0ed-b38e302afe1d-model-cache\") pod \"s3-tls-init-serving-h9vjj\" (UID: \"6f75400d-a054-447b-a0ed-b38e302afe1d\") " pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:12.975833 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.975776    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v78lf\" (UniqueName: \"kubernetes.io/projected/6f75400d-a054-447b-a0ed-b38e302afe1d-kube-api-access-v78lf\") pod \"s3-tls-init-serving-h9vjj\" (UID: \"6f75400d-a054-447b-a0ed-b38e302afe1d\") " pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:12.976097 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.976076    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f75400d-a054-447b-a0ed-b38e302afe1d-model-cache\") pod \"s3-tls-init-serving-h9vjj\" (UID: \"6f75400d-a054-447b-a0ed-b38e302afe1d\") " pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:12.984677 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:12.984642    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v78lf\" (UniqueName: \"kubernetes.io/projected/6f75400d-a054-447b-a0ed-b38e302afe1d-kube-api-access-v78lf\") pod \"s3-tls-init-serving-h9vjj\" (UID: \"6f75400d-a054-447b-a0ed-b38e302afe1d\") " pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:13.074276 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:13.074185    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:13.353238 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:13.353071    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-h9vjj"]
Sep 17 16:06:13.356203 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:06:13.356170    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f75400d_a054_447b_a0ed_b38e302afe1d.slice/crio-c369bdd84f502bf68bf4dc27ca63d0fe0c35cfff00321a43e318351eb04a8ff7 WatchSource:0}: Error finding container c369bdd84f502bf68bf4dc27ca63d0fe0c35cfff00321a43e318351eb04a8ff7: Status 404 returned error can't find the container with id c369bdd84f502bf68bf4dc27ca63d0fe0c35cfff00321a43e318351eb04a8ff7
Sep 17 16:06:13.987581 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:13.987546    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-h9vjj" event={"ID":"6f75400d-a054-447b-a0ed-b38e302afe1d","Type":"ContainerStarted","Data":"d658567e2460cf3f166d2b3045a0a335d04a48bf5f4f0492d900ac25267bd376"}
Sep 17 16:06:13.987581 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:13.987588    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-h9vjj" event={"ID":"6f75400d-a054-447b-a0ed-b38e302afe1d","Type":"ContainerStarted","Data":"c369bdd84f502bf68bf4dc27ca63d0fe0c35cfff00321a43e318351eb04a8ff7"}
Sep 17 16:06:15.997811 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:15.997775    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f75400d-a054-447b-a0ed-b38e302afe1d" containerID="d658567e2460cf3f166d2b3045a0a335d04a48bf5f4f0492d900ac25267bd376" exitCode=0
Sep 17 16:06:15.998332 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:15.997854    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-h9vjj" event={"ID":"6f75400d-a054-447b-a0ed-b38e302afe1d","Type":"ContainerDied","Data":"d658567e2460cf3f166d2b3045a0a335d04a48bf5f4f0492d900ac25267bd376"}
Sep 17 16:06:17.003990 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:17.003954    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-h9vjj" event={"ID":"6f75400d-a054-447b-a0ed-b38e302afe1d","Type":"ContainerStarted","Data":"fa68eddd4e6e5049fecc265118fd6f12ed2d91b925708437a436a6967031cd33"}
Sep 17 16:06:17.026497 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:17.026443    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-h9vjj" podStartSLOduration=3.283346084 podStartE2EDuration="5.026427486s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.743081402s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:17.026005995 +0000 UTC m=+743.291520994" watchObservedRunningTime="2026-09-17 16:06:17.026427486 +0000 UTC m=+743.291942485"
Sep 17 16:06:23.034574 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:23.034527    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f75400d-a054-447b-a0ed-b38e302afe1d" containerID="fa68eddd4e6e5049fecc265118fd6f12ed2d91b925708437a436a6967031cd33" exitCode=0
Sep 17 16:06:23.035048 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:23.034606    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-h9vjj" event={"ID":"6f75400d-a054-447b-a0ed-b38e302afe1d","Type":"ContainerDied","Data":"fa68eddd4e6e5049fecc265118fd6f12ed2d91b925708437a436a6967031cd33"}
Sep 17 16:06:24.189565 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:24.189537    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:24.280265 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:24.280229    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f75400d-a054-447b-a0ed-b38e302afe1d-model-cache\") pod \"6f75400d-a054-447b-a0ed-b38e302afe1d\" (UID: \"6f75400d-a054-447b-a0ed-b38e302afe1d\") "
Sep 17 16:06:24.280518 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:24.280345    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v78lf\" (UniqueName: \"kubernetes.io/projected/6f75400d-a054-447b-a0ed-b38e302afe1d-kube-api-access-v78lf\") pod \"6f75400d-a054-447b-a0ed-b38e302afe1d\" (UID: \"6f75400d-a054-447b-a0ed-b38e302afe1d\") "
Sep 17 16:06:24.282753 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:24.282707    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f75400d-a054-447b-a0ed-b38e302afe1d-kube-api-access-v78lf" (OuterVolumeSpecName: "kube-api-access-v78lf") pod "6f75400d-a054-447b-a0ed-b38e302afe1d" (UID: "6f75400d-a054-447b-a0ed-b38e302afe1d"). InnerVolumeSpecName "kube-api-access-v78lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:06:24.304071 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:24.303989    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f75400d-a054-447b-a0ed-b38e302afe1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f75400d-a054-447b-a0ed-b38e302afe1d" (UID: "6f75400d-a054-447b-a0ed-b38e302afe1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:06:24.381764 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:24.381728    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v78lf\" (UniqueName: \"kubernetes.io/projected/6f75400d-a054-447b-a0ed-b38e302afe1d-kube-api-access-v78lf\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:24.381764 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:24.381762    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f75400d-a054-447b-a0ed-b38e302afe1d-model-cache\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:25.047322 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:25.047260    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-h9vjj" event={"ID":"6f75400d-a054-447b-a0ed-b38e302afe1d","Type":"ContainerDied","Data":"c369bdd84f502bf68bf4dc27ca63d0fe0c35cfff00321a43e318351eb04a8ff7"}
Sep 17 16:06:25.047322 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:25.047322    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-h9vjj"
Sep 17 16:06:25.047540 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:25.047327    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c369bdd84f502bf68bf4dc27ca63d0fe0c35cfff00321a43e318351eb04a8ff7"
Sep 17 16:06:38.460875 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.460842    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"]
Sep 17 16:06:38.461329 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.461143    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f75400d-a054-447b-a0ed-b38e302afe1d" containerName="s3-tls-init-serving"
Sep 17 16:06:38.461329 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.461154    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f75400d-a054-447b-a0ed-b38e302afe1d" containerName="s3-tls-init-serving"
Sep 17 16:06:38.461329 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.461168    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f75400d-a054-447b-a0ed-b38e302afe1d" containerName="download-hf-model"
Sep 17 16:06:38.461329 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.461173    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f75400d-a054-447b-a0ed-b38e302afe1d" containerName="download-hf-model"
Sep 17 16:06:38.461329 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.461235    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f75400d-a054-447b-a0ed-b38e302afe1d" containerName="s3-tls-init-serving"
Sep 17 16:06:38.465949 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.465933    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.479842 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.479819    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:06:38.479958 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.479917    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 16:06:38.480028 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.479977    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:06:38.480087 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.480037    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:06:38.480557 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.480533    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:06:38.480657 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.480538    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 16:06:38.481421 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.481407    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:06:38.492586 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.492565    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"]
Sep 17 16:06:38.494362 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.494341    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.494472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.494380    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbb7\" (UniqueName: \"kubernetes.io/projected/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-api-access-gwbb7\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.494472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.494422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.494472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.494463    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.494628 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.494555    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.595242 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.595211    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.595242 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.595250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbb7\" (UniqueName: \"kubernetes.io/projected/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-api-access-gwbb7\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.595510 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.595301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.595510 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.595349    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.595510 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.595406    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.595665 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:06:38.595527    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 17 16:06:38.595665 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:06:38.595587    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls podName:78e56b0f-6f39-45af-b0c7-a946266a2b4c nodeName:}" failed. No retries permitted until 2026-09-17 16:06:39.09556565 +0000 UTC m=+765.361080642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls") pod "isvc-sklearn-batcher-predictor-75db54674f-rvw5s" (UID: "78e56b0f-6f39-45af-b0c7-a946266a2b4c") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 17 16:06:38.595780 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.595690    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.596220 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.596200    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.596705 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.596683    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.610152 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.610130    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbb7\" (UniqueName: \"kubernetes.io/projected/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-api-access-gwbb7\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:38.667901 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.667877    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"]
Sep 17 16:06:38.671774 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.671760    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.674313 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.674276    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e8cd39\""
Sep 17 16:06:38.674468 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.674443    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e8cd39-predictor-serving-cert\""
Sep 17 16:06:38.674574 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.674479    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e8cd39-kube-rbac-proxy-sar-config\""
Sep 17 16:06:38.674574 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.674508    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e8cd39-dockercfg-b4php\""
Sep 17 16:06:38.686414 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.686394    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"]
Sep 17 16:06:38.696079 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.696058    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-cabundle-cert\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.696181 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.696093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfl9\" (UniqueName: \"kubernetes.io/projected/57caa384-cd14-461b-8f33-60c821a47fde-kube-api-access-plfl9\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.696181 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.696113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.696181 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.696149    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.696181 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.696180    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57caa384-cd14-461b-8f33-60c821a47fde-kserve-provision-location\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.797084 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.797041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.797321 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.797095    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57caa384-cd14-461b-8f33-60c821a47fde-kserve-provision-location\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.797321 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.797216    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-cabundle-cert\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.797321 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.797250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfl9\" (UniqueName: \"kubernetes.io/projected/57caa384-cd14-461b-8f33-60c821a47fde-kube-api-access-plfl9\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.797321 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:06:38.797220    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-serving-cert: secret "isvc-init-fail-e8cd39-predictor-serving-cert" not found
Sep 17 16:06:38.797321 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.797278    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.797590 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:06:38.797386    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls podName:57caa384-cd14-461b-8f33-60c821a47fde nodeName:}" failed. No retries permitted until 2026-09-17 16:06:39.297362073 +0000 UTC m=+765.562877071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls") pod "isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj" (UID: "57caa384-cd14-461b-8f33-60c821a47fde") : secret "isvc-init-fail-e8cd39-predictor-serving-cert" not found
Sep 17 16:06:38.797590 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.797524    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57caa384-cd14-461b-8f33-60c821a47fde-kserve-provision-location\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.797933 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.797911    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-cabundle-cert\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.798114 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.798093    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:38.806708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:38.806685    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfl9\" (UniqueName: \"kubernetes.io/projected/57caa384-cd14-461b-8f33-60c821a47fde-kube-api-access-plfl9\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:39.099574 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:39.099480    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:39.101996 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:39.101966    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-75db54674f-rvw5s\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:39.300431 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:39.300395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:39.302877 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:39.302856    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls\") pod \"isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") " pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:39.375967 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:39.375857    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:06:39.583190 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:39.583150    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:39.660231 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:39.660200    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"]
Sep 17 16:06:39.666780 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:06:39.666741    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e56b0f_6f39_45af_b0c7_a946266a2b4c.slice/crio-aaf6d21c21d622091d0069f5cbaa8d65b3d9cc8dde89792e979a5aa1e07dcbe3 WatchSource:0}: Error finding container aaf6d21c21d622091d0069f5cbaa8d65b3d9cc8dde89792e979a5aa1e07dcbe3: Status 404 returned error can't find the container with id aaf6d21c21d622091d0069f5cbaa8d65b3d9cc8dde89792e979a5aa1e07dcbe3
Sep 17 16:06:39.870640 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:39.866725    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"]
Sep 17 16:06:40.113176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:40.113135    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerStarted","Data":"291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497"}
Sep 17 16:06:40.113393 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:40.113223    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerStarted","Data":"aaf6d21c21d622091d0069f5cbaa8d65b3d9cc8dde89792e979a5aa1e07dcbe3"}
Sep 17 16:06:40.114961 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:40.114935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj" event={"ID":"57caa384-cd14-461b-8f33-60c821a47fde","Type":"ContainerStarted","Data":"816d2a69ebaa8b0b647f624f4be8e3caccbb7aa11c63932d3a576e2a69648561"}
Sep 17 16:06:40.115102 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:40.114987    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj" event={"ID":"57caa384-cd14-461b-8f33-60c821a47fde","Type":"ContainerStarted","Data":"d8d4dd105fcf0385e9568a6d2068d15c10832c30299890606b31b93779f33b4f"}
Sep 17 16:06:44.135075 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:44.135034    2585 generic.go:356] "Generic (PLEG): container finished" podID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerID="291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497" exitCode=0
Sep 17 16:06:44.135511 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:44.135089    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerDied","Data":"291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497"}
Sep 17 16:06:45.154217 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:45.154060    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj_57caa384-cd14-461b-8f33-60c821a47fde/storage-initializer/0.log"
Sep 17 16:06:45.154217 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:45.154106    2585 generic.go:356] "Generic (PLEG): container finished" podID="57caa384-cd14-461b-8f33-60c821a47fde" containerID="816d2a69ebaa8b0b647f624f4be8e3caccbb7aa11c63932d3a576e2a69648561" exitCode=1
Sep 17 16:06:45.154217 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:45.154222    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj" event={"ID":"57caa384-cd14-461b-8f33-60c821a47fde","Type":"ContainerDied","Data":"816d2a69ebaa8b0b647f624f4be8e3caccbb7aa11c63932d3a576e2a69648561"}
Sep 17 16:06:46.167129 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:46.167096    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj_57caa384-cd14-461b-8f33-60c821a47fde/storage-initializer/0.log"
Sep 17 16:06:46.167633 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:46.167173    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj" event={"ID":"57caa384-cd14-461b-8f33-60c821a47fde","Type":"ContainerStarted","Data":"a749941e27944c5df6bc8f387c897a3396f4eae6872a3d389d0e7fb5055355ac"}
Sep 17 16:06:48.260870 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:48.260805    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"]
Sep 17 16:06:48.261480 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:48.261072    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj" podUID="57caa384-cd14-461b-8f33-60c821a47fde" containerName="storage-initializer" containerID="cri-o://a749941e27944c5df6bc8f387c897a3396f4eae6872a3d389d0e7fb5055355ac" gracePeriod=30
Sep 17 16:06:53.475850 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.475798    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"]
Sep 17 16:06:53.487186 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.487148    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.490204 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.490179    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 16:06:53.492857 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.492828    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 16:06:53.498893 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.498811    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"]
Sep 17 16:06:53.526597 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.526554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7gf\" (UniqueName: \"kubernetes.io/projected/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-api-access-gd7gf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.526775 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.526649    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.526775 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.526719    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9418521f-d7a7-44b4-a363-bf69b2f21d02-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.526775 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.526759    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.526899 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.526816    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.627522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.627489    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.627661 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.627547    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9418521f-d7a7-44b4-a363-bf69b2f21d02-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.627661 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.627579    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.627661 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.627603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.627661 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.627634    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7gf\" (UniqueName: \"kubernetes.io/projected/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-api-access-gd7gf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.627860 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:06:53.627677    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 17 16:06:53.627860 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:06:53.627762    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls podName:9418521f-d7a7-44b4-a363-bf69b2f21d02 nodeName:}" failed. No retries permitted until 2026-09-17 16:06:54.12774111 +0000 UTC m=+780.393256092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" (UID: "9418521f-d7a7-44b4-a363-bf69b2f21d02") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 17 16:06:53.628112 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.628076    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9418521f-d7a7-44b4-a363-bf69b2f21d02-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.628383 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.628362    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.628707 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.628680    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:53.651947 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:53.651911    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7gf\" (UniqueName: \"kubernetes.io/projected/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-api-access-gd7gf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:54.133577 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:54.133538    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:54.136571 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:54.136528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cf6v4\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:54.215581 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:54.215552    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj_57caa384-cd14-461b-8f33-60c821a47fde/storage-initializer/1.log"
Sep 17 16:06:54.216539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:54.216060    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj_57caa384-cd14-461b-8f33-60c821a47fde/storage-initializer/0.log"
Sep 17 16:06:54.216539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:54.216099    2585 generic.go:356] "Generic (PLEG): container finished" podID="57caa384-cd14-461b-8f33-60c821a47fde" containerID="a749941e27944c5df6bc8f387c897a3396f4eae6872a3d389d0e7fb5055355ac" exitCode=1
Sep 17 16:06:54.216539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:54.216234    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj" event={"ID":"57caa384-cd14-461b-8f33-60c821a47fde","Type":"ContainerDied","Data":"a749941e27944c5df6bc8f387c897a3396f4eae6872a3d389d0e7fb5055355ac"}
Sep 17 16:06:54.216539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:54.216320    2585 scope.go:117] "RemoveContainer" containerID="816d2a69ebaa8b0b647f624f4be8e3caccbb7aa11c63932d3a576e2a69648561"
Sep 17 16:06:54.400643 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:54.400530    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:06:58.167916 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.167889    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj_57caa384-cd14-461b-8f33-60c821a47fde/storage-initializer/1.log"
Sep 17 16:06:58.169992 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.169973    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:58.244593 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.244563    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj_57caa384-cd14-461b-8f33-60c821a47fde/storage-initializer/1.log"
Sep 17 16:06:58.244782 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.244623    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj" event={"ID":"57caa384-cd14-461b-8f33-60c821a47fde","Type":"ContainerDied","Data":"d8d4dd105fcf0385e9568a6d2068d15c10832c30299890606b31b93779f33b4f"}
Sep 17 16:06:58.244782 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.244681    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"
Sep 17 16:06:58.244782 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.244692    2585 scope.go:117] "RemoveContainer" containerID="a749941e27944c5df6bc8f387c897a3396f4eae6872a3d389d0e7fb5055355ac"
Sep 17 16:06:58.272625 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.272602    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plfl9\" (UniqueName: \"kubernetes.io/projected/57caa384-cd14-461b-8f33-60c821a47fde-kube-api-access-plfl9\") pod \"57caa384-cd14-461b-8f33-60c821a47fde\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") "
Sep 17 16:06:58.272737 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.272677    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-cabundle-cert\") pod \"57caa384-cd14-461b-8f33-60c821a47fde\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") "
Sep 17 16:06:58.272737 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.272710    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-kube-rbac-proxy-sar-config\") pod \"57caa384-cd14-461b-8f33-60c821a47fde\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") "
Sep 17 16:06:58.272737 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.272726    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls\") pod \"57caa384-cd14-461b-8f33-60c821a47fde\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") "
Sep 17 16:06:58.272892 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.272879    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57caa384-cd14-461b-8f33-60c821a47fde-kserve-provision-location\") pod \"57caa384-cd14-461b-8f33-60c821a47fde\" (UID: \"57caa384-cd14-461b-8f33-60c821a47fde\") "
Sep 17 16:06:58.273271 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.273237    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57caa384-cd14-461b-8f33-60c821a47fde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57caa384-cd14-461b-8f33-60c821a47fde" (UID: "57caa384-cd14-461b-8f33-60c821a47fde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:06:58.273403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.273388    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57caa384-cd14-461b-8f33-60c821a47fde" (UID: "57caa384-cd14-461b-8f33-60c821a47fde"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:58.273560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.273495    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57caa384-cd14-461b-8f33-60c821a47fde" (UID: "57caa384-cd14-461b-8f33-60c821a47fde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:58.275070 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.275045    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57caa384-cd14-461b-8f33-60c821a47fde" (UID: "57caa384-cd14-461b-8f33-60c821a47fde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:06:58.275548 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.275511    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57caa384-cd14-461b-8f33-60c821a47fde-kube-api-access-plfl9" (OuterVolumeSpecName: "kube-api-access-plfl9") pod "57caa384-cd14-461b-8f33-60c821a47fde" (UID: "57caa384-cd14-461b-8f33-60c821a47fde"). InnerVolumeSpecName "kube-api-access-plfl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:06:58.374482 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.374438    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:58.374482 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.374473    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57caa384-cd14-461b-8f33-60c821a47fde-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:58.374482 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.374483    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57caa384-cd14-461b-8f33-60c821a47fde-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:58.374482 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.374492    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57caa384-cd14-461b-8f33-60c821a47fde-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:58.374769 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.374502    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plfl9\" (UniqueName: \"kubernetes.io/projected/57caa384-cd14-461b-8f33-60c821a47fde-kube-api-access-plfl9\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:06:58.424888 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.424858    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"]
Sep 17 16:06:58.426781 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:06:58.426747    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9418521f_d7a7_44b4_a363_bf69b2f21d02.slice/crio-d4064217df790b64e33cd1f3c4aa9d51620f11ce4cf230c0fd9b2478c8cf782c WatchSource:0}: Error finding container d4064217df790b64e33cd1f3c4aa9d51620f11ce4cf230c0fd9b2478c8cf782c: Status 404 returned error can't find the container with id d4064217df790b64e33cd1f3c4aa9d51620f11ce4cf230c0fd9b2478c8cf782c
Sep 17 16:06:58.592356 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.592321    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"]
Sep 17 16:06:58.599406 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:58.599371    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e8cd39-predictor-7cc55f569c-47hfj"]
Sep 17 16:06:59.251450 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:59.251412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerStarted","Data":"5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd"}
Sep 17 16:06:59.253768 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:59.253737    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" event={"ID":"9418521f-d7a7-44b4-a363-bf69b2f21d02","Type":"ContainerStarted","Data":"106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0"}
Sep 17 16:06:59.253934 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:59.253788    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" event={"ID":"9418521f-d7a7-44b4-a363-bf69b2f21d02","Type":"ContainerStarted","Data":"d4064217df790b64e33cd1f3c4aa9d51620f11ce4cf230c0fd9b2478c8cf782c"}
Sep 17 16:06:59.782714 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:06:59.782684    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57caa384-cd14-461b-8f33-60c821a47fde" path="/var/lib/kubelet/pods/57caa384-cd14-461b-8f33-60c821a47fde/volumes"
Sep 17 16:07:03.283948 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:03.283915    2585 generic.go:356] "Generic (PLEG): container finished" podID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerID="106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0" exitCode=0
Sep 17 16:07:03.284429 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:03.283994    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" event={"ID":"9418521f-d7a7-44b4-a363-bf69b2f21d02","Type":"ContainerDied","Data":"106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0"}
Sep 17 16:07:03.286857 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:03.286829    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerStarted","Data":"44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c"}
Sep 17 16:07:16.378168 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:16.378122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerStarted","Data":"80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe"}
Sep 17 16:07:16.379115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:16.378817    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:07:16.379115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:16.378878    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:07:16.379115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:16.379060    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:07:16.380863 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:16.380819    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:07:16.381613 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:16.381589    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:16.385780 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:16.385759    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:07:16.411987 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:16.411931    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podStartSLOduration=3.588229604 podStartE2EDuration="38.411913247s" totalImagesPullingTime="31.074442376s" totalInitContainerRuntime="3.749241267s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:16.409185385 +0000 UTC m=+802.674700434" watchObservedRunningTime="2026-09-17 16:07:16.411913247 +0000 UTC m=+802.677428246"
Sep 17 16:07:17.385757 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:17.385712    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:07:17.386250 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:17.386042    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:18.392152 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:18.391773    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:07:18.392152 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:18.391839    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:28.391533 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:28.391472    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:07:28.392017 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:28.391984    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:28.448786 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:28.448743    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" event={"ID":"9418521f-d7a7-44b4-a363-bf69b2f21d02","Type":"ContainerStarted","Data":"070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265"}
Sep 17 16:07:28.448945 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:28.448839    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" event={"ID":"9418521f-d7a7-44b4-a363-bf69b2f21d02","Type":"ContainerStarted","Data":"10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b"}
Sep 17 16:07:28.449063 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:28.449042    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:07:28.472659 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:28.472598    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podStartSLOduration=7.005828549 podStartE2EDuration="35.472578272s" totalImagesPullingTime="24.530752557s" totalInitContainerRuntime="3.935997166s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:28.470739159 +0000 UTC m=+814.736254172" watchObservedRunningTime="2026-09-17 16:07:28.472578272 +0000 UTC m=+814.738093269"
Sep 17 16:07:29.453187 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:29.453144    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:07:29.454440 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:29.454419    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:07:30.457747 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:30.457705    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:07:35.462770 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:35.462743    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:07:35.463238 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:35.463210    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:07:38.391584 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:38.391534    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:07:38.391999 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:38.391933    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:45.463695 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:45.463655    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:07:48.392039 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:48.391987    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:07:48.392577 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:48.392552    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:55.463611 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:55.463569    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:07:58.392115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:58.392061    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:07:58.392631 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:07:58.392574    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:05.464033 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:05.463991    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:08:08.392202 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:08.392129    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:08:08.392624 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:08.392600    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:15.463506 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:15.463466    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:08:18.391700 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:18.391652    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:08:18.392158 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:18.392091    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:25.464147 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:25.464106    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:08:28.392605 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:28.392566    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:08:28.393028 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:28.392715    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:08:35.463178 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:35.463135    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:08:43.411068 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.411031    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"]
Sep 17 16:08:43.411604 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.411407    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" containerID="cri-o://80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe" gracePeriod=30
Sep 17 16:08:43.411604 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.411420    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" containerID="cri-o://5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd" gracePeriod=30
Sep 17 16:08:43.411604 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.411426    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy" containerID="cri-o://44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c" gracePeriod=30
Sep 17 16:08:43.496684 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.496637    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"]
Sep 17 16:08:43.497004 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.496991    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57caa384-cd14-461b-8f33-60c821a47fde" containerName="storage-initializer"
Sep 17 16:08:43.497004 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.497005    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="57caa384-cd14-461b-8f33-60c821a47fde" containerName="storage-initializer"
Sep 17 16:08:43.497091 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.497076    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="57caa384-cd14-461b-8f33-60c821a47fde" containerName="storage-initializer"
Sep 17 16:08:43.497139 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.497129    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57caa384-cd14-461b-8f33-60c821a47fde" containerName="storage-initializer"
Sep 17 16:08:43.497173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.497138    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="57caa384-cd14-461b-8f33-60c821a47fde" containerName="storage-initializer"
Sep 17 16:08:43.497206 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.497199    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="57caa384-cd14-461b-8f33-60c821a47fde" containerName="storage-initializer"
Sep 17 16:08:43.500344 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.500327    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.503041 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.503013    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 16:08:43.503041 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.503028    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 16:08:43.517662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.517641    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"]
Sep 17 16:08:43.629126 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.629085    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e0b3f5a-f0de-4f6c-854e-77d763372723-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.629126 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.629124    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.629417 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.629160    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e0b3f5a-f0de-4f6c-854e-77d763372723-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.629417 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.629201    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.629417 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.629263    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s66j\" (UniqueName: \"kubernetes.io/projected/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-api-access-7s66j\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.730156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.730049    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e0b3f5a-f0de-4f6c-854e-77d763372723-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.730156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.730109    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.730427 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.730178    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s66j\" (UniqueName: \"kubernetes.io/projected/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-api-access-7s66j\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.730427 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.730238    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e0b3f5a-f0de-4f6c-854e-77d763372723-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.730427 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.730265    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.730590 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.730512    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e0b3f5a-f0de-4f6c-854e-77d763372723-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.730881 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.730858    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.731043 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.731024    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.733036 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.733015    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e0b3f5a-f0de-4f6c-854e-77d763372723-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.739325 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.739301    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s66j\" (UniqueName: \"kubernetes.io/projected/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-api-access-7s66j\") pod \"isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:43.792602 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.792566    2585 generic.go:356] "Generic (PLEG): container finished" podID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerID="44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c" exitCode=2
Sep 17 16:08:43.792775 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.792635    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerDied","Data":"44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c"}
Sep 17 16:08:43.810856 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:43.810817    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:44.088641 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:44.088609    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"]
Sep 17 16:08:44.091967 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:08:44.091941    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0b3f5a_f0de_4f6c_854e_77d763372723.slice/crio-5fbb8daecd67e04ee26217329d779727e1cee8734b43597dfc898c822dafe73c WatchSource:0}: Error finding container 5fbb8daecd67e04ee26217329d779727e1cee8734b43597dfc898c822dafe73c: Status 404 returned error can't find the container with id 5fbb8daecd67e04ee26217329d779727e1cee8734b43597dfc898c822dafe73c
Sep 17 16:08:44.778171 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:44.778122    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:08:44.801856 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:44.801819    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerStarted","Data":"c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44"}
Sep 17 16:08:44.802010 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:44.801884    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerStarted","Data":"5fbb8daecd67e04ee26217329d779727e1cee8734b43597dfc898c822dafe73c"}
Sep 17 16:08:46.379310 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:46.379247    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 16:08:48.392448 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:48.392392    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:08:48.392933 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:48.392765    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:48.823563 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:48.823525    2585 generic.go:356] "Generic (PLEG): container finished" podID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerID="c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44" exitCode=0
Sep 17 16:08:48.823723 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:48.823595    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerDied","Data":"c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44"}
Sep 17 16:08:48.826460 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:48.826436    2585 generic.go:356] "Generic (PLEG): container finished" podID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerID="5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd" exitCode=0
Sep 17 16:08:48.826522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:48.826494    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerDied","Data":"5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd"}
Sep 17 16:08:49.833690 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:49.833651    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerStarted","Data":"4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85"}
Sep 17 16:08:49.834070 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:49.833734    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerStarted","Data":"bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb"}
Sep 17 16:08:49.834070 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:49.833760    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerStarted","Data":"b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c"}
Sep 17 16:08:49.834070 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:49.834053    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:49.834245 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:49.834122    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:49.835590 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:49.835564    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:08:49.857502 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:49.857454    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podStartSLOduration=3.015619523 podStartE2EDuration="6.857438276s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841818753s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:49.855102051 +0000 UTC m=+896.120617050" watchObservedRunningTime="2026-09-17 16:08:49.857438276 +0000 UTC m=+896.122953276"
Sep 17 16:08:50.838432 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:50.838399    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:50.838900 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:50.838568    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:08:50.839818 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:50.839791    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:51.379520 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:51.379475    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 16:08:51.843224 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:51.843176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:08:51.843734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:51.843682    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:54.779515 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:54.779472    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:08:55.754107 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:55.754076    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:08:55.756698 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:55.756674    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:08:56.380255 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:56.380214    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 16:08:56.380702 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:56.380347    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:08:56.848807 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:56.848780    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:08:56.849352 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:56.849323    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:08:56.849631 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:56.849598    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:58.391759 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:58.391707    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:08:58.392164 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:08:58.392000    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:01.379981 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:01.379934    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 16:09:04.189775 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:04.189742    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"]
Sep 17 16:09:04.190152 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:04.189981    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" containerID="cri-o://10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b" gracePeriod=30
Sep 17 16:09:04.190152 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:04.190023    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kube-rbac-proxy" containerID="cri-o://070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265" gracePeriod=30
Sep 17 16:09:04.778050 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:04.778007    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:09:04.907148 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:04.907113    2585 generic.go:356] "Generic (PLEG): container finished" podID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerID="070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265" exitCode=2
Sep 17 16:09:04.907148 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:04.907151    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" event={"ID":"9418521f-d7a7-44b4-a363-bf69b2f21d02","Type":"ContainerDied","Data":"070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265"}
Sep 17 16:09:05.458940 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:05.458894    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 16:09:06.379904 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:06.379861    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 16:09:06.849647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:06.849602    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:09:06.850193 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:06.850170    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:08.391616 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:08.391568    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:09:08.392058 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:08.391709    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:09:08.392058 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:08.391819    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:08.392058 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:08.391942    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:09:09.638212 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.638181    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:09:09.661379 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.661346    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9418521f-d7a7-44b4-a363-bf69b2f21d02-kserve-provision-location\") pod \"9418521f-d7a7-44b4-a363-bf69b2f21d02\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") "
Sep 17 16:09:09.661379 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.661380    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-cabundle-cert\") pod \"9418521f-d7a7-44b4-a363-bf69b2f21d02\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") "
Sep 17 16:09:09.661642 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.661407    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-rbac-proxy-sar-config\") pod \"9418521f-d7a7-44b4-a363-bf69b2f21d02\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") "
Sep 17 16:09:09.661642 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.661452    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls\") pod \"9418521f-d7a7-44b4-a363-bf69b2f21d02\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") "
Sep 17 16:09:09.661859 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.661808    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9418521f-d7a7-44b4-a363-bf69b2f21d02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9418521f-d7a7-44b4-a363-bf69b2f21d02" (UID: "9418521f-d7a7-44b4-a363-bf69b2f21d02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:09:09.661859 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.661827    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9418521f-d7a7-44b4-a363-bf69b2f21d02" (UID: "9418521f-d7a7-44b4-a363-bf69b2f21d02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:09:09.661859 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.661825    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9418521f-d7a7-44b4-a363-bf69b2f21d02" (UID: "9418521f-d7a7-44b4-a363-bf69b2f21d02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:09:09.664351 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.664318    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9418521f-d7a7-44b4-a363-bf69b2f21d02" (UID: "9418521f-d7a7-44b4-a363-bf69b2f21d02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:09:09.762559 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.762527    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd7gf\" (UniqueName: \"kubernetes.io/projected/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-api-access-gd7gf\") pod \"9418521f-d7a7-44b4-a363-bf69b2f21d02\" (UID: \"9418521f-d7a7-44b4-a363-bf69b2f21d02\") "
Sep 17 16:09:09.762741 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.762729    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9418521f-d7a7-44b4-a363-bf69b2f21d02-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:09.762786 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.762745    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9418521f-d7a7-44b4-a363-bf69b2f21d02-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:09.762786 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.762754    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:09.762786 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.762764    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:09.764780 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.764739    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-api-access-gd7gf" (OuterVolumeSpecName: "kube-api-access-gd7gf") pod "9418521f-d7a7-44b4-a363-bf69b2f21d02" (UID: "9418521f-d7a7-44b4-a363-bf69b2f21d02"). InnerVolumeSpecName "kube-api-access-gd7gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:09:09.863368 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.863331    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd7gf\" (UniqueName: \"kubernetes.io/projected/9418521f-d7a7-44b4-a363-bf69b2f21d02-kube-api-access-gd7gf\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:09.932346 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.932308    2585 generic.go:356] "Generic (PLEG): container finished" podID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerID="10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b" exitCode=0
Sep 17 16:09:09.932507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.932366    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"
Sep 17 16:09:09.932507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.932377    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" event={"ID":"9418521f-d7a7-44b4-a363-bf69b2f21d02","Type":"ContainerDied","Data":"10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b"}
Sep 17 16:09:09.932507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.932438    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4" event={"ID":"9418521f-d7a7-44b4-a363-bf69b2f21d02","Type":"ContainerDied","Data":"d4064217df790b64e33cd1f3c4aa9d51620f11ce4cf230c0fd9b2478c8cf782c"}
Sep 17 16:09:09.932507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.932464    2585 scope.go:117] "RemoveContainer" containerID="070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265"
Sep 17 16:09:09.941240 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.941217    2585 scope.go:117] "RemoveContainer" containerID="10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b"
Sep 17 16:09:09.952316 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.952163    2585 scope.go:117] "RemoveContainer" containerID="106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0"
Sep 17 16:09:09.953819 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.953798    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"]
Sep 17 16:09:09.958662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.958625    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cf6v4"]
Sep 17 16:09:09.962335 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.962315    2585 scope.go:117] "RemoveContainer" containerID="070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265"
Sep 17 16:09:09.962714 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:09:09.962692    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265\": container with ID starting with 070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265 not found: ID does not exist" containerID="070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265"
Sep 17 16:09:09.962798 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.962724    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265"} err="failed to get container status \"070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265\": rpc error: code = NotFound desc = could not find container \"070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265\": container with ID starting with 070e9af516167ec540c5f5af56bfa55d53a22e9f9d7345ec7c34074e2b017265 not found: ID does not exist"
Sep 17 16:09:09.962798 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.962744    2585 scope.go:117] "RemoveContainer" containerID="10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b"
Sep 17 16:09:09.963055 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:09:09.963035    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b\": container with ID starting with 10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b not found: ID does not exist" containerID="10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b"
Sep 17 16:09:09.963104 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.963064    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b"} err="failed to get container status \"10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b\": rpc error: code = NotFound desc = could not find container \"10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b\": container with ID starting with 10a9bead460bdad6a2199f1fdd0ad1284819a66668d3acb41e3da428065fbf0b not found: ID does not exist"
Sep 17 16:09:09.963104 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.963082    2585 scope.go:117] "RemoveContainer" containerID="106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0"
Sep 17 16:09:09.963387 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:09:09.963366    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0\": container with ID starting with 106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0 not found: ID does not exist" containerID="106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0"
Sep 17 16:09:09.963459 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:09.963390    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0"} err="failed to get container status \"106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0\": rpc error: code = NotFound desc = could not find container \"106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0\": container with ID starting with 106c9a016ec158f7a8ea4754a761ddb9233a08742ec6d086a08ea92ec00d62c0 not found: ID does not exist"
Sep 17 16:09:11.379392 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:11.379344    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 16:09:11.782395 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:11.782362    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" path="/var/lib/kubelet/pods/9418521f-d7a7-44b4-a363-bf69b2f21d02/volumes"
Sep 17 16:09:13.567300 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.567264    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:09:13.589127 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589096    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwbb7\" (UniqueName: \"kubernetes.io/projected/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-api-access-gwbb7\") pod \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") "
Sep 17 16:09:13.589328 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589138    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kserve-provision-location\") pod \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") "
Sep 17 16:09:13.589328 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589165    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-cabundle-cert\") pod \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") "
Sep 17 16:09:13.589328 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589195    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-rbac-proxy-sar-config\") pod \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") "
Sep 17 16:09:13.589328 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589315    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls\") pod \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\" (UID: \"78e56b0f-6f39-45af-b0c7-a946266a2b4c\") "
Sep 17 16:09:13.589558 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589506    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78e56b0f-6f39-45af-b0c7-a946266a2b4c" (UID: "78e56b0f-6f39-45af-b0c7-a946266a2b4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:09:13.589558 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589550    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "78e56b0f-6f39-45af-b0c7-a946266a2b4c" (UID: "78e56b0f-6f39-45af-b0c7-a946266a2b4c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:09:13.589652 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589594    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78e56b0f-6f39-45af-b0c7-a946266a2b4c" (UID: "78e56b0f-6f39-45af-b0c7-a946266a2b4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:09:13.589652 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589620    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:13.589652 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.589634    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:13.591700 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.591677    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-api-access-gwbb7" (OuterVolumeSpecName: "kube-api-access-gwbb7") pod "78e56b0f-6f39-45af-b0c7-a946266a2b4c" (UID: "78e56b0f-6f39-45af-b0c7-a946266a2b4c"). InnerVolumeSpecName "kube-api-access-gwbb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:09:13.591700 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.591687    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78e56b0f-6f39-45af-b0c7-a946266a2b4c" (UID: "78e56b0f-6f39-45af-b0c7-a946266a2b4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:09:13.690961 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.690870    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:13.690961 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.690904    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e56b0f-6f39-45af-b0c7-a946266a2b4c-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:13.690961 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.690923    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwbb7\" (UniqueName: \"kubernetes.io/projected/78e56b0f-6f39-45af-b0c7-a946266a2b4c-kube-api-access-gwbb7\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:09:13.956930 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.956832    2585 generic.go:356] "Generic (PLEG): container finished" podID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerID="80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe" exitCode=0
Sep 17 16:09:13.956930 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.956896    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"
Sep 17 16:09:13.956930 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.956907    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerDied","Data":"80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe"}
Sep 17 16:09:13.957151 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.956978    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s" event={"ID":"78e56b0f-6f39-45af-b0c7-a946266a2b4c","Type":"ContainerDied","Data":"aaf6d21c21d622091d0069f5cbaa8d65b3d9cc8dde89792e979a5aa1e07dcbe3"}
Sep 17 16:09:13.957151 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.957009    2585 scope.go:117] "RemoveContainer" containerID="80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe"
Sep 17 16:09:13.965826 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.965808    2585 scope.go:117] "RemoveContainer" containerID="44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c"
Sep 17 16:09:13.973834 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.973815    2585 scope.go:117] "RemoveContainer" containerID="5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd"
Sep 17 16:09:13.975418 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.975397    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"]
Sep 17 16:09:13.979081 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.979056    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75db54674f-rvw5s"]
Sep 17 16:09:13.982129 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.982116    2585 scope.go:117] "RemoveContainer" containerID="291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497"
Sep 17 16:09:13.989676 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.989655    2585 scope.go:117] "RemoveContainer" containerID="80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe"
Sep 17 16:09:13.989979 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:09:13.989954    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe\": container with ID starting with 80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe not found: ID does not exist" containerID="80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe"
Sep 17 16:09:13.990092 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.989986    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe"} err="failed to get container status \"80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe\": rpc error: code = NotFound desc = could not find container \"80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe\": container with ID starting with 80fd8453fc6b99d17e8ab34bae6947d8e3b76c3171760fecc3c0ba04fa1a03fe not found: ID does not exist"
Sep 17 16:09:13.990092 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.990012    2585 scope.go:117] "RemoveContainer" containerID="44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c"
Sep 17 16:09:13.990311 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:09:13.990275    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c\": container with ID starting with 44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c not found: ID does not exist" containerID="44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c"
Sep 17 16:09:13.990358 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.990322    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c"} err="failed to get container status \"44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c\": rpc error: code = NotFound desc = could not find container \"44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c\": container with ID starting with 44294247069e274430c8ca4e499e33efc0faa694750bc4c1cbba87f4b7de3e1c not found: ID does not exist"
Sep 17 16:09:13.990358 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.990344    2585 scope.go:117] "RemoveContainer" containerID="5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd"
Sep 17 16:09:13.990605 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:09:13.990588    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd\": container with ID starting with 5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd not found: ID does not exist" containerID="5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd"
Sep 17 16:09:13.990652 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.990611    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd"} err="failed to get container status \"5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd\": rpc error: code = NotFound desc = could not find container \"5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd\": container with ID starting with 5c903cccd702f31f603cc26d9edbdc217b6077543de293c90695a439a3825fbd not found: ID does not exist"
Sep 17 16:09:13.990652 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.990627    2585 scope.go:117] "RemoveContainer" containerID="291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497"
Sep 17 16:09:13.990885 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:09:13.990866    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497\": container with ID starting with 291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497 not found: ID does not exist" containerID="291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497"
Sep 17 16:09:13.990937 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:13.990892    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497"} err="failed to get container status \"291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497\": rpc error: code = NotFound desc = could not find container \"291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497\": container with ID starting with 291ec410abbe4d30976e07626f1745f87c2c858847b1e1e40022fa828d1f9497 not found: ID does not exist"
Sep 17 16:09:15.783363 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:15.783325    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" path="/var/lib/kubelet/pods/78e56b0f-6f39-45af-b0c7-a946266a2b4c/volumes"
Sep 17 16:09:16.849930 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:16.849893    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:09:16.850428 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:16.850406    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:26.849343 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:26.849269    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:09:26.849926 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:26.849807    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:36.849747 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:36.849702    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:09:36.852179 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:36.850068    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:46.849864 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:46.849815    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:09:46.850321 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:46.850234    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:56.849462 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:56.849416    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:09:56.849960 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:09:56.849832    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:06.850451 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:06.850363    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:10:06.850932 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:06.850647    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:10:18.554751 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:18.554709    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"]
Sep 17 16:10:18.555197 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:18.555107    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" containerID="cri-o://b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c" gracePeriod=30
Sep 17 16:10:18.555197 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:18.555121    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" containerID="cri-o://4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85" gracePeriod=30
Sep 17 16:10:18.555197 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:18.555143    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy" containerID="cri-o://bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb" gracePeriod=30
Sep 17 16:10:19.269312 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:19.269260    2585 generic.go:356] "Generic (PLEG): container finished" podID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerID="bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb" exitCode=2
Sep 17 16:10:19.269494 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:19.269323    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerDied","Data":"bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb"}
Sep 17 16:10:21.843708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:21.843665    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 16:10:23.290163 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:23.290132    2585 generic.go:356] "Generic (PLEG): container finished" podID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerID="b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c" exitCode=0
Sep 17 16:10:23.290533 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:23.290201    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerDied","Data":"b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c"}
Sep 17 16:10:26.843571 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:26.843529    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 16:10:26.849992 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:26.849954    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:10:26.850323 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:26.850305    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:31.843649 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:31.843597    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 16:10:31.844077 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:31.843752    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:10:36.844035 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:36.843986    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 16:10:36.849495 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:36.849455    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:10:36.849863 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:36.849834    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:38.867971 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.867932    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"]
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868260    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868272    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868308    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kube-rbac-proxy"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868317    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kube-rbac-proxy"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868327    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="storage-initializer"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868333    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="storage-initializer"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868343    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868348    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868356    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="storage-initializer"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868361    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="storage-initializer"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868366    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868371    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868378    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868383    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container"
Sep 17 16:10:38.868430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868438    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="agent"
Sep 17 16:10:38.868895 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868446    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kube-rbac-proxy"
Sep 17 16:10:38.868895 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868456    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kube-rbac-proxy"
Sep 17 16:10:38.868895 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868461    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9418521f-d7a7-44b4-a363-bf69b2f21d02" containerName="kserve-container"
Sep 17 16:10:38.868895 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.868468    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e56b0f-6f39-45af-b0c7-a946266a2b4c" containerName="kserve-container"
Sep 17 16:10:38.871727 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.871710    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:38.881240 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.881216    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 16:10:38.881386 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:38.881214    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 16:10:39.006914 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.006872    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"]
Sep 17 16:10:39.029308 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.029261    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.029509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.029383    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6edace48-e622-493c-9c52-fd38d1748508-kserve-provision-location\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.029509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.029412    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-cabundle-cert\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.029509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.029442    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6edace48-e622-493c-9c52-fd38d1748508-proxy-tls\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.029509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.029470    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4xc\" (UniqueName: \"kubernetes.io/projected/6edace48-e622-493c-9c52-fd38d1748508-kube-api-access-pg4xc\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.130505 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.130406    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6edace48-e622-493c-9c52-fd38d1748508-kserve-provision-location\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.130505 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.130450    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-cabundle-cert\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.130505 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.130483    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6edace48-e622-493c-9c52-fd38d1748508-proxy-tls\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.130505 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.130502    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4xc\" (UniqueName: \"kubernetes.io/projected/6edace48-e622-493c-9c52-fd38d1748508-kube-api-access-pg4xc\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.130820 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.130537    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.130866 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.130853    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6edace48-e622-493c-9c52-fd38d1748508-kserve-provision-location\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.131198 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.131177    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-cabundle-cert\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.131198 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.131204    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.133127 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.133103    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6edace48-e622-493c-9c52-fd38d1748508-proxy-tls\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.169663 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.169627    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4xc\" (UniqueName: \"kubernetes.io/projected/6edace48-e622-493c-9c52-fd38d1748508-kube-api-access-pg4xc\") pod \"isvc-logger-predictor-59b8454b74-lj88s\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.182306 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.182250    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:39.475539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:39.475504    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"]
Sep 17 16:10:39.476368 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:10:39.476343    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edace48_e622_493c_9c52_fd38d1748508.slice/crio-7a446941801689ef9373b6804105b676f38fcafbf248df44f23516561950db12 WatchSource:0}: Error finding container 7a446941801689ef9373b6804105b676f38fcafbf248df44f23516561950db12: Status 404 returned error can't find the container with id 7a446941801689ef9373b6804105b676f38fcafbf248df44f23516561950db12
Sep 17 16:10:40.367696 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:40.367662    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerStarted","Data":"641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b"}
Sep 17 16:10:40.368194 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:40.367723    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerStarted","Data":"7a446941801689ef9373b6804105b676f38fcafbf248df44f23516561950db12"}
Sep 17 16:10:41.843626 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:41.843582    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 16:10:43.383659 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:43.383623    2585 generic.go:356] "Generic (PLEG): container finished" podID="6edace48-e622-493c-9c52-fd38d1748508" containerID="641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b" exitCode=0
Sep 17 16:10:43.384036 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:43.383689    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerDied","Data":"641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b"}
Sep 17 16:10:44.391804 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.391767    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerStarted","Data":"125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72"}
Sep 17 16:10:44.392247 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.391843    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerStarted","Data":"0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1"}
Sep 17 16:10:44.392247 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.391871    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerStarted","Data":"cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be"}
Sep 17 16:10:44.392247 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.392175    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:44.392247 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.392240    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:44.392495 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.392297    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:44.393839 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.393810    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:10:44.394517 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.394493    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:44.420782 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:44.420730    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podStartSLOduration=2.607195206 podStartE2EDuration="6.420709151s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.813513945s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:44.418729978 +0000 UTC m=+1010.684244980" watchObservedRunningTime="2026-09-17 16:10:44.420709151 +0000 UTC m=+1010.686224150"
Sep 17 16:10:45.396728 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:45.396667    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:10:45.397216 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:45.397121    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:46.843568 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:46.843527    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 16:10:46.850022 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:46.849985    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 16:10:46.850173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:46.850151    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:10:46.850383 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:46.850360    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:46.850484 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:46.850470    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:10:48.705000 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.704968    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:10:48.717442 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.717412    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e0b3f5a-f0de-4f6c-854e-77d763372723-kserve-provision-location\") pod \"8e0b3f5a-f0de-4f6c-854e-77d763372723\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") "
Sep 17 16:10:48.717442 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.717448    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-cabundle-cert\") pod \"8e0b3f5a-f0de-4f6c-854e-77d763372723\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") "
Sep 17 16:10:48.717657 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.717487    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-rbac-proxy-sar-config\") pod \"8e0b3f5a-f0de-4f6c-854e-77d763372723\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") "
Sep 17 16:10:48.717657 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.717515    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s66j\" (UniqueName: \"kubernetes.io/projected/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-api-access-7s66j\") pod \"8e0b3f5a-f0de-4f6c-854e-77d763372723\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") "
Sep 17 16:10:48.717657 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.717547    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e0b3f5a-f0de-4f6c-854e-77d763372723-proxy-tls\") pod \"8e0b3f5a-f0de-4f6c-854e-77d763372723\" (UID: \"8e0b3f5a-f0de-4f6c-854e-77d763372723\") "
Sep 17 16:10:48.717851 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.717767    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0b3f5a-f0de-4f6c-854e-77d763372723-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e0b3f5a-f0de-4f6c-854e-77d763372723" (UID: "8e0b3f5a-f0de-4f6c-854e-77d763372723"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:48.717919 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.717902    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e0b3f5a-f0de-4f6c-854e-77d763372723" (UID: "8e0b3f5a-f0de-4f6c-854e-77d763372723"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:48.717972 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.717918    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e0b3f5a-f0de-4f6c-854e-77d763372723" (UID: "8e0b3f5a-f0de-4f6c-854e-77d763372723"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:48.719877 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.719849    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0b3f5a-f0de-4f6c-854e-77d763372723-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e0b3f5a-f0de-4f6c-854e-77d763372723" (UID: "8e0b3f5a-f0de-4f6c-854e-77d763372723"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:10:48.719980 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.719876    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-api-access-7s66j" (OuterVolumeSpecName: "kube-api-access-7s66j") pod "8e0b3f5a-f0de-4f6c-854e-77d763372723" (UID: "8e0b3f5a-f0de-4f6c-854e-77d763372723"). InnerVolumeSpecName "kube-api-access-7s66j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:10:48.818472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.818370    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s66j\" (UniqueName: \"kubernetes.io/projected/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-api-access-7s66j\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:10:48.818472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.818415    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e0b3f5a-f0de-4f6c-854e-77d763372723-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:10:48.818472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.818427    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e0b3f5a-f0de-4f6c-854e-77d763372723-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:10:48.818472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.818435    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:10:48.818472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:48.818445    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e0b3f5a-f0de-4f6c-854e-77d763372723-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:10:49.417215 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.417173    2585 generic.go:356] "Generic (PLEG): container finished" podID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerID="4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85" exitCode=0
Sep 17 16:10:49.417384 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.417318    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"
Sep 17 16:10:49.417384 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.417318    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerDied","Data":"4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85"}
Sep 17 16:10:49.417485 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.417384    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw" event={"ID":"8e0b3f5a-f0de-4f6c-854e-77d763372723","Type":"ContainerDied","Data":"5fbb8daecd67e04ee26217329d779727e1cee8734b43597dfc898c822dafe73c"}
Sep 17 16:10:49.417485 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.417415    2585 scope.go:117] "RemoveContainer" containerID="4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85"
Sep 17 16:10:49.427063 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.427041    2585 scope.go:117] "RemoveContainer" containerID="bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb"
Sep 17 16:10:49.435561 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.435541    2585 scope.go:117] "RemoveContainer" containerID="b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c"
Sep 17 16:10:49.442786 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.442759    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"]
Sep 17 16:10:49.444101 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.444078    2585 scope.go:117] "RemoveContainer" containerID="c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44"
Sep 17 16:10:49.450172 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.450145    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8c4795cdc-mvtpw"]
Sep 17 16:10:49.453236 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.453217    2585 scope.go:117] "RemoveContainer" containerID="4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85"
Sep 17 16:10:49.453576 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:10:49.453559    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85\": container with ID starting with 4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85 not found: ID does not exist" containerID="4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85"
Sep 17 16:10:49.453639 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.453585    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85"} err="failed to get container status \"4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85\": rpc error: code = NotFound desc = could not find container \"4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85\": container with ID starting with 4ce7c83ae4afb32a85f3e8989ace3cf74e1b76996233b6b2920f47857f896f85 not found: ID does not exist"
Sep 17 16:10:49.453639 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.453604    2585 scope.go:117] "RemoveContainer" containerID="bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb"
Sep 17 16:10:49.453882 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:10:49.453860    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb\": container with ID starting with bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb not found: ID does not exist" containerID="bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb"
Sep 17 16:10:49.453929 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.453890    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb"} err="failed to get container status \"bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb\": rpc error: code = NotFound desc = could not find container \"bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb\": container with ID starting with bfeef0c1c4b253af41629407818977638fcc8e9332ab79d91a5a4063156083cb not found: ID does not exist"
Sep 17 16:10:49.453929 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.453906    2585 scope.go:117] "RemoveContainer" containerID="b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c"
Sep 17 16:10:49.454217 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:10:49.454197    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c\": container with ID starting with b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c not found: ID does not exist" containerID="b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c"
Sep 17 16:10:49.454392 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.454230    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c"} err="failed to get container status \"b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c\": rpc error: code = NotFound desc = could not find container \"b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c\": container with ID starting with b4d0e0a77fdf85240d809140bd7a68379c6f33b31948ffcdbe8bb6e139668f0c not found: ID does not exist"
Sep 17 16:10:49.454392 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.454247    2585 scope.go:117] "RemoveContainer" containerID="c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44"
Sep 17 16:10:49.454564 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:10:49.454549    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44\": container with ID starting with c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44 not found: ID does not exist" containerID="c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44"
Sep 17 16:10:49.454607 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.454567    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44"} err="failed to get container status \"c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44\": rpc error: code = NotFound desc = could not find container \"c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44\": container with ID starting with c0caebfd99baf7f0e7d785ee1cf02627a97f488d35fb04f079f2c77e2ead9e44 not found: ID does not exist"
Sep 17 16:10:49.782169 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:49.782134    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" path="/var/lib/kubelet/pods/8e0b3f5a-f0de-4f6c-854e-77d763372723/volumes"
Sep 17 16:10:50.400894 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:50.400861    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:10:50.401427 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:50.401398    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:10:50.401738 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:10:50.401716    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:11:00.401859 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:00.401816    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:11:00.402652 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:00.402624    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:11:10.401949 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:10.401905    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:11:10.402463 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:10.402439    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:11:20.402048 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:20.402000    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:11:20.402537 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:20.402406    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:11:30.401817 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:30.401763    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:11:30.402348 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:30.402176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:11:40.401446 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:40.401401    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:11:40.401942 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:40.401852    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:11:50.401669 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:50.401622    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:11:50.402186 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:50.402129    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:11:58.777927 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:58.777894    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:11:58.778423 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:11:58.778078    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:12:13.905453 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:13.905416    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"]
Sep 17 16:12:13.906059 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:13.905696    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" containerID="cri-o://125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72" gracePeriod=30
Sep 17 16:12:13.906059 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:13.905681    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy" containerID="cri-o://0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1" gracePeriod=30
Sep 17 16:12:13.906059 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:13.905806    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" containerID="cri-o://cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be" gracePeriod=30
Sep 17 16:12:14.791998 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:14.791961    2585 generic.go:356] "Generic (PLEG): container finished" podID="6edace48-e622-493c-9c52-fd38d1748508" containerID="0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1" exitCode=2
Sep 17 16:12:14.792167 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:14.792024    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerDied","Data":"0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1"}
Sep 17 16:12:15.396873 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:15.396831    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:12:18.777370 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:18.777324    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:12:18.777814 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:18.777686    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:18.811150 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:18.811064    2585 generic.go:356] "Generic (PLEG): container finished" podID="6edace48-e622-493c-9c52-fd38d1748508" containerID="cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be" exitCode=0
Sep 17 16:12:18.811150 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:18.811112    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerDied","Data":"cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be"}
Sep 17 16:12:20.397126 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:20.397078    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:12:25.397464 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:25.397415    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:12:25.397853 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:25.397594    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:12:28.777470 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:28.777422    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:12:28.777935 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:28.777769    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:30.397325 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:30.397254    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:12:35.397082 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:35.397033    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:12:38.777727 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:38.777683    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:12:38.778166 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:38.777822    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:12:38.778166 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:38.778019    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:38.778166 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:38.778129    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:12:40.397681 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:40.397634    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:12:44.060248 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.060224    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:12:44.113955 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.113926    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-kube-rbac-proxy-sar-config\") pod \"6edace48-e622-493c-9c52-fd38d1748508\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") "
Sep 17 16:12:44.113955 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.113963    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg4xc\" (UniqueName: \"kubernetes.io/projected/6edace48-e622-493c-9c52-fd38d1748508-kube-api-access-pg4xc\") pod \"6edace48-e622-493c-9c52-fd38d1748508\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") "
Sep 17 16:12:44.114174 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114101    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6edace48-e622-493c-9c52-fd38d1748508-kserve-provision-location\") pod \"6edace48-e622-493c-9c52-fd38d1748508\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") "
Sep 17 16:12:44.114174 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114151    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-cabundle-cert\") pod \"6edace48-e622-493c-9c52-fd38d1748508\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") "
Sep 17 16:12:44.114307 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114182    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6edace48-e622-493c-9c52-fd38d1748508-proxy-tls\") pod \"6edace48-e622-493c-9c52-fd38d1748508\" (UID: \"6edace48-e622-493c-9c52-fd38d1748508\") "
Sep 17 16:12:44.114370 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114308    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6edace48-e622-493c-9c52-fd38d1748508" (UID: "6edace48-e622-493c-9c52-fd38d1748508"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:12:44.114484 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114460    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edace48-e622-493c-9c52-fd38d1748508-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6edace48-e622-493c-9c52-fd38d1748508" (UID: "6edace48-e622-493c-9c52-fd38d1748508"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:12:44.114548 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114508    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6edace48-e622-493c-9c52-fd38d1748508" (UID: "6edace48-e622-493c-9c52-fd38d1748508"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:12:44.114604 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114585    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:12:44.114640 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114607    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6edace48-e622-493c-9c52-fd38d1748508-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:12:44.114640 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.114619    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6edace48-e622-493c-9c52-fd38d1748508-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:12:44.116255 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.116235    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edace48-e622-493c-9c52-fd38d1748508-kube-api-access-pg4xc" (OuterVolumeSpecName: "kube-api-access-pg4xc") pod "6edace48-e622-493c-9c52-fd38d1748508" (UID: "6edace48-e622-493c-9c52-fd38d1748508"). InnerVolumeSpecName "kube-api-access-pg4xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:12:44.116420 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.116398    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edace48-e622-493c-9c52-fd38d1748508-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6edace48-e622-493c-9c52-fd38d1748508" (UID: "6edace48-e622-493c-9c52-fd38d1748508"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:12:44.215411 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.215332    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6edace48-e622-493c-9c52-fd38d1748508-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:12:44.215411 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.215359    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg4xc\" (UniqueName: \"kubernetes.io/projected/6edace48-e622-493c-9c52-fd38d1748508-kube-api-access-pg4xc\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:12:44.926147 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.926110    2585 generic.go:356] "Generic (PLEG): container finished" podID="6edace48-e622-493c-9c52-fd38d1748508" containerID="125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72" exitCode=0
Sep 17 16:12:44.926344 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.926175    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"
Sep 17 16:12:44.926344 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.926200    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerDied","Data":"125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72"}
Sep 17 16:12:44.926344 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.926305    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s" event={"ID":"6edace48-e622-493c-9c52-fd38d1748508","Type":"ContainerDied","Data":"7a446941801689ef9373b6804105b676f38fcafbf248df44f23516561950db12"}
Sep 17 16:12:44.926463 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.926352    2585 scope.go:117] "RemoveContainer" containerID="125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72"
Sep 17 16:12:44.935034 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.935017    2585 scope.go:117] "RemoveContainer" containerID="0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1"
Sep 17 16:12:44.947993 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.947969    2585 scope.go:117] "RemoveContainer" containerID="cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be"
Sep 17 16:12:44.956234 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.956205    2585 scope.go:117] "RemoveContainer" containerID="641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b"
Sep 17 16:12:44.956430 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.956405    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"]
Sep 17 16:12:44.959935 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.959913    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-59b8454b74-lj88s"]
Sep 17 16:12:44.964296 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.964262    2585 scope.go:117] "RemoveContainer" containerID="125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72"
Sep 17 16:12:44.964579 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:12:44.964562    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72\": container with ID starting with 125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72 not found: ID does not exist" containerID="125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72"
Sep 17 16:12:44.964661 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.964586    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72"} err="failed to get container status \"125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72\": rpc error: code = NotFound desc = could not find container \"125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72\": container with ID starting with 125eb66ac45f8ada7d1254620194a15a26e785723ac2aab61c3a9405f64f4d72 not found: ID does not exist"
Sep 17 16:12:44.964661 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.964603    2585 scope.go:117] "RemoveContainer" containerID="0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1"
Sep 17 16:12:44.964895 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:12:44.964880    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1\": container with ID starting with 0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1 not found: ID does not exist" containerID="0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1"
Sep 17 16:12:44.964942 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.964897    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1"} err="failed to get container status \"0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1\": rpc error: code = NotFound desc = could not find container \"0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1\": container with ID starting with 0bfc2f058b875e69fc35dfb716b2ad5aca255eebbec3ad1f73781f9f0656c2a1 not found: ID does not exist"
Sep 17 16:12:44.964942 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.964909    2585 scope.go:117] "RemoveContainer" containerID="cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be"
Sep 17 16:12:44.965140 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:12:44.965120    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be\": container with ID starting with cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be not found: ID does not exist" containerID="cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be"
Sep 17 16:12:44.965205 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.965147    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be"} err="failed to get container status \"cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be\": rpc error: code = NotFound desc = could not find container \"cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be\": container with ID starting with cfa2a3d893d637e8793444ce23f715eab75dc5b6df1f7748ac5c3bdd6b5ab9be not found: ID does not exist"
Sep 17 16:12:44.965205 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.965164    2585 scope.go:117] "RemoveContainer" containerID="641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b"
Sep 17 16:12:44.965451 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:12:44.965420    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b\": container with ID starting with 641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b not found: ID does not exist" containerID="641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b"
Sep 17 16:12:44.965497 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:44.965454    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b"} err="failed to get container status \"641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b\": rpc error: code = NotFound desc = could not find container \"641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b\": container with ID starting with 641a15ff61578656d87a867ce8da4fbf8983c777a126e3b0fff329a7a9132a3b not found: ID does not exist"
Sep 17 16:12:45.782350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:12:45.782318    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6edace48-e622-493c-9c52-fd38d1748508" path="/var/lib/kubelet/pods/6edace48-e622-493c-9c52-fd38d1748508/volumes"
Sep 17 16:13:01.462961 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.462922    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"]
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463232    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463243    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463256    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463262    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463270    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463275    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463297    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463302    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463309    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463314    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463321    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463328    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463332    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="storage-initializer"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463338    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="storage-initializer"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463351    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="storage-initializer"
Sep 17 16:13:01.463371 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463357    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="storage-initializer"
Sep 17 16:13:01.463844 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463402    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kserve-container"
Sep 17 16:13:01.463844 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463410    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="agent"
Sep 17 16:13:01.463844 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463416    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kserve-container"
Sep 17 16:13:01.463844 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463422    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="agent"
Sep 17 16:13:01.463844 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463428    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e0b3f5a-f0de-4f6c-854e-77d763372723" containerName="kube-rbac-proxy"
Sep 17 16:13:01.463844 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.463434    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edace48-e622-493c-9c52-fd38d1748508" containerName="kube-rbac-proxy"
Sep 17 16:13:01.467890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.467874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.470601 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.470575    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:13:01.470742 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.470645    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:13:01.470742 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.470576    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:13:01.470742 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.470706    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:13:01.470956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.470855    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 16:13:01.470956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.470904    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:13:01.471517 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.471499    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:13:01.482994 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.482970    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"]
Sep 17 16:13:01.557764 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.557723    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6skpl\" (UniqueName: \"kubernetes.io/projected/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-api-access-6skpl\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.557947 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.557843    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5524140a-6e0e-481a-90e1-3ec562e45c62-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.558029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.557945    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.558029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.557989    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.558029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.558021    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.659265 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.659223    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5524140a-6e0e-481a-90e1-3ec562e45c62-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.659481 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.659310    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.659481 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.659350    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.659481 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.659369    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.659481 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.659413    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6skpl\" (UniqueName: \"kubernetes.io/projected/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-api-access-6skpl\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.659703 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:13:01.659514    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 16:13:01.659703 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:13:01.659588    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls podName:5524140a-6e0e-481a-90e1-3ec562e45c62 nodeName:}" failed. No retries permitted until 2026-09-17 16:13:02.159564539 +0000 UTC m=+1148.425079541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" (UID: "5524140a-6e0e-481a-90e1-3ec562e45c62") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 16:13:01.659703 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.659692    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5524140a-6e0e-481a-90e1-3ec562e45c62-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.660090 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.660065    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.660136 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.660095    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:01.674587 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:01.674557    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6skpl\" (UniqueName: \"kubernetes.io/projected/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-api-access-6skpl\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:02.164313 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:02.164225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:02.166841 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:02.166817    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:02.378432 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:02.378389    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:02.656079 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:02.656054    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"]
Sep 17 16:13:02.658867 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:13:02.658822    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5524140a_6e0e_481a_90e1_3ec562e45c62.slice/crio-e83c20ba94c2d0cb920e24c8e45da22d5aa3b928f056d7c1ed5c39085689deef WatchSource:0}: Error finding container e83c20ba94c2d0cb920e24c8e45da22d5aa3b928f056d7c1ed5c39085689deef: Status 404 returned error can't find the container with id e83c20ba94c2d0cb920e24c8e45da22d5aa3b928f056d7c1ed5c39085689deef
Sep 17 16:13:03.015785 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:03.015750    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" event={"ID":"5524140a-6e0e-481a-90e1-3ec562e45c62","Type":"ContainerStarted","Data":"71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2"}
Sep 17 16:13:03.015975 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:03.015816    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" event={"ID":"5524140a-6e0e-481a-90e1-3ec562e45c62","Type":"ContainerStarted","Data":"e83c20ba94c2d0cb920e24c8e45da22d5aa3b928f056d7c1ed5c39085689deef"}
Sep 17 16:13:07.033040 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:07.033003    2585 generic.go:356] "Generic (PLEG): container finished" podID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerID="71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2" exitCode=0
Sep 17 16:13:07.033440 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:07.033074    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" event={"ID":"5524140a-6e0e-481a-90e1-3ec562e45c62","Type":"ContainerDied","Data":"71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2"}
Sep 17 16:13:08.039585 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:08.039550    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" event={"ID":"5524140a-6e0e-481a-90e1-3ec562e45c62","Type":"ContainerStarted","Data":"3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf"}
Sep 17 16:13:08.040005 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:08.039614    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" event={"ID":"5524140a-6e0e-481a-90e1-3ec562e45c62","Type":"ContainerStarted","Data":"c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b"}
Sep 17 16:13:08.040005 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:08.039896    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:08.063559 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:08.063511    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podStartSLOduration=3.4107916129999998 podStartE2EDuration="7.063493945s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.652702332s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:13:08.062026767 +0000 UTC m=+1154.327541766" watchObservedRunningTime="2026-09-17 16:13:08.063493945 +0000 UTC m=+1154.329008942"
Sep 17 16:13:09.044756 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:09.044708    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:15.055025 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:15.054992    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:13:45.055828 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:45.055786    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:13:55.055833 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:55.055788    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:13:55.792599 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:55.792565    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:13:55.793116 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:13:55.793098    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:14:05.056365 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:05.056323    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:14:15.055921 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:15.055885    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:14:25.640929 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:25.640897    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:14:31.581472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:31.581438    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"]
Sep 17 16:14:31.581907 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:31.581670    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container" containerID="cri-o://c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b" gracePeriod=30
Sep 17 16:14:31.581907 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:31.581727    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kube-rbac-proxy" containerID="cri-o://3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf" gracePeriod=30
Sep 17 16:14:32.398548 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:32.398514    2585 generic.go:356] "Generic (PLEG): container finished" podID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerID="3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf" exitCode=2
Sep 17 16:14:32.398723 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:32.398587    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" event={"ID":"5524140a-6e0e-481a-90e1-3ec562e45c62","Type":"ContainerDied","Data":"3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf"}
Sep 17 16:14:35.051018 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:35.050974    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 16:14:35.056390 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:35.056360    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:14:36.827368 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.827329    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"]
Sep 17 16:14:36.830852 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.830824    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.833572 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.833543    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 16:14:36.833726 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.833544    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:14:36.843175 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.843142    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"]
Sep 17 16:14:36.882828 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.882789    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c96b261-795b-40a0-a084-c021bde0b5b7-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.882828 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.882833    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jszpp\" (UniqueName: \"kubernetes.io/projected/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-api-access-jszpp\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.883139 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.882914    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.883139 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.882966    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c96b261-795b-40a0-a084-c021bde0b5b7-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.883139 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.883014    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.939608 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.939583    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:14:36.984066 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.984019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c96b261-795b-40a0-a084-c021bde0b5b7-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.984066 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.984065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jszpp\" (UniqueName: \"kubernetes.io/projected/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-api-access-jszpp\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.984354 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.984108    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.984354 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.984130    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c96b261-795b-40a0-a084-c021bde0b5b7-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.984354 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.984159    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.984612 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.984585    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c96b261-795b-40a0-a084-c021bde0b5b7-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.984838 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.984819    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.984912 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.984878    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.986813 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.986794    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c96b261-795b-40a0-a084-c021bde0b5b7-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:36.992621 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:36.992591    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jszpp\" (UniqueName: \"kubernetes.io/projected/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-api-access-jszpp\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:37.085063 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085023    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-rbac-proxy-sar-config\") pod \"5524140a-6e0e-481a-90e1-3ec562e45c62\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") "
Sep 17 16:14:37.085266 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085073    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls\") pod \"5524140a-6e0e-481a-90e1-3ec562e45c62\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") "
Sep 17 16:14:37.085266 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085093    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6skpl\" (UniqueName: \"kubernetes.io/projected/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-api-access-6skpl\") pod \"5524140a-6e0e-481a-90e1-3ec562e45c62\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") "
Sep 17 16:14:37.085266 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085124    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5524140a-6e0e-481a-90e1-3ec562e45c62-kserve-provision-location\") pod \"5524140a-6e0e-481a-90e1-3ec562e45c62\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") "
Sep 17 16:14:37.085266 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085249    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-cabundle-cert\") pod \"5524140a-6e0e-481a-90e1-3ec562e45c62\" (UID: \"5524140a-6e0e-481a-90e1-3ec562e45c62\") "
Sep 17 16:14:37.085545 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085422    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5524140a-6e0e-481a-90e1-3ec562e45c62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5524140a-6e0e-481a-90e1-3ec562e45c62" (UID: "5524140a-6e0e-481a-90e1-3ec562e45c62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:14:37.085589 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085554    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5524140a-6e0e-481a-90e1-3ec562e45c62" (UID: "5524140a-6e0e-481a-90e1-3ec562e45c62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:14:37.085626 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085588    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5524140a-6e0e-481a-90e1-3ec562e45c62" (UID: "5524140a-6e0e-481a-90e1-3ec562e45c62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:14:37.085662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085640    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5524140a-6e0e-481a-90e1-3ec562e45c62-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:14:37.085662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.085653    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:14:37.087522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.087499    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5524140a-6e0e-481a-90e1-3ec562e45c62" (UID: "5524140a-6e0e-481a-90e1-3ec562e45c62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:14:37.087606 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.087559    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-api-access-6skpl" (OuterVolumeSpecName: "kube-api-access-6skpl") pod "5524140a-6e0e-481a-90e1-3ec562e45c62" (UID: "5524140a-6e0e-481a-90e1-3ec562e45c62"). InnerVolumeSpecName "kube-api-access-6skpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:14:37.142351 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.142274    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:37.186853 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.186815    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5524140a-6e0e-481a-90e1-3ec562e45c62-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:14:37.186853 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.186850    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6skpl\" (UniqueName: \"kubernetes.io/projected/5524140a-6e0e-481a-90e1-3ec562e45c62-kube-api-access-6skpl\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:14:37.186853 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.186860    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5524140a-6e0e-481a-90e1-3ec562e45c62-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:14:37.419007 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.418986    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"]
Sep 17 16:14:37.420621 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.420590    2585 generic.go:356] "Generic (PLEG): container finished" podID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerID="c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b" exitCode=0
Sep 17 16:14:37.420734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.420656    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" event={"ID":"5524140a-6e0e-481a-90e1-3ec562e45c62","Type":"ContainerDied","Data":"c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b"}
Sep 17 16:14:37.420734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.420686    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"
Sep 17 16:14:37.420734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.420702    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l" event={"ID":"5524140a-6e0e-481a-90e1-3ec562e45c62","Type":"ContainerDied","Data":"e83c20ba94c2d0cb920e24c8e45da22d5aa3b928f056d7c1ed5c39085689deef"}
Sep 17 16:14:37.420734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.420729    2585 scope.go:117] "RemoveContainer" containerID="3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf"
Sep 17 16:14:37.421262 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:14:37.421239    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c96b261_795b_40a0_a084_c021bde0b5b7.slice/crio-d20adb2c96d46149527436d82234eda1494fae800f0715fc2bc5796b58d706f3 WatchSource:0}: Error finding container d20adb2c96d46149527436d82234eda1494fae800f0715fc2bc5796b58d706f3: Status 404 returned error can't find the container with id d20adb2c96d46149527436d82234eda1494fae800f0715fc2bc5796b58d706f3
Sep 17 16:14:37.429838 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.429644    2585 scope.go:117] "RemoveContainer" containerID="c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b"
Sep 17 16:14:37.438611 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.438588    2585 scope.go:117] "RemoveContainer" containerID="71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2"
Sep 17 16:14:37.447476 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.447452    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"]
Sep 17 16:14:37.447635 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.447611    2585 scope.go:117] "RemoveContainer" containerID="3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf"
Sep 17 16:14:37.447871 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:14:37.447854    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf\": container with ID starting with 3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf not found: ID does not exist" containerID="3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf"
Sep 17 16:14:37.447918 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.447877    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf"} err="failed to get container status \"3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf\": rpc error: code = NotFound desc = could not find container \"3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf\": container with ID starting with 3f434622ff3f80e39fe48e7a83aaade3e3a6a372a62d7abd05f92ec83e25b6bf not found: ID does not exist"
Sep 17 16:14:37.447918 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.447894    2585 scope.go:117] "RemoveContainer" containerID="c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b"
Sep 17 16:14:37.448115 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:14:37.448096    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b\": container with ID starting with c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b not found: ID does not exist" containerID="c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b"
Sep 17 16:14:37.448235 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.448124    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b"} err="failed to get container status \"c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b\": rpc error: code = NotFound desc = could not find container \"c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b\": container with ID starting with c37d7e442dd721f4d1092140a020ec65b73db97b99579961f989ae495230cc5b not found: ID does not exist"
Sep 17 16:14:37.448235 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.448147    2585 scope.go:117] "RemoveContainer" containerID="71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2"
Sep 17 16:14:37.448465 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:14:37.448449    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2\": container with ID starting with 71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2 not found: ID does not exist" containerID="71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2"
Sep 17 16:14:37.448505 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.448468    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2"} err="failed to get container status \"71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2\": rpc error: code = NotFound desc = could not find container \"71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2\": container with ID starting with 71e32d6530fd1ac398a555e059cc408dedb18212947dd9261b4e26f50d6fd7c2 not found: ID does not exist"
Sep 17 16:14:37.450635 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.450614    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-b5j5l"]
Sep 17 16:14:37.782541 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:37.782510    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" path="/var/lib/kubelet/pods/5524140a-6e0e-481a-90e1-3ec562e45c62/volumes"
Sep 17 16:14:38.427082 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:38.427046    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" event={"ID":"8c96b261-795b-40a0-a084-c021bde0b5b7","Type":"ContainerStarted","Data":"ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c"}
Sep 17 16:14:38.427582 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:38.427106    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" event={"ID":"8c96b261-795b-40a0-a084-c021bde0b5b7","Type":"ContainerStarted","Data":"d20adb2c96d46149527436d82234eda1494fae800f0715fc2bc5796b58d706f3"}
Sep 17 16:14:41.442901 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:41.442874    2585 generic.go:356] "Generic (PLEG): container finished" podID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerID="ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c" exitCode=0
Sep 17 16:14:41.443208 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:41.442953    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" event={"ID":"8c96b261-795b-40a0-a084-c021bde0b5b7","Type":"ContainerDied","Data":"ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c"}
Sep 17 16:14:42.455200 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:42.455164    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" event={"ID":"8c96b261-795b-40a0-a084-c021bde0b5b7","Type":"ContainerStarted","Data":"5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d"}
Sep 17 16:14:42.455606 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:42.455245    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" event={"ID":"8c96b261-795b-40a0-a084-c021bde0b5b7","Type":"ContainerStarted","Data":"ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c"}
Sep 17 16:14:42.455690 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:42.455599    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:42.455690 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:42.455660    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:14:42.479645 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:42.479592    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podStartSLOduration=2.5786411620000003 podStartE2EDuration="6.479574907s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.900933745s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:14:42.478635304 +0000 UTC m=+1248.744150304" watchObservedRunningTime="2026-09-17 16:14:42.479574907 +0000 UTC m=+1248.745089909"
Sep 17 16:14:48.464843 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:14:48.464810    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:15:18.466155 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:15:18.466110    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 16:15:28.466025 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:15:28.465981    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 16:15:38.465846 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:15:38.465799    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 16:15:48.466019 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:15:48.465963    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 16:15:59.041099 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:15:59.041066    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:16:06.830115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:06.830033    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"]
Sep 17 16:16:06.830629 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:06.830367    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container" containerID="cri-o://ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c" gracePeriod=30
Sep 17 16:16:06.830629 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:06.830386    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kube-rbac-proxy" containerID="cri-o://5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d" gracePeriod=30
Sep 17 16:16:07.828446 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:07.828410    2585 generic.go:356] "Generic (PLEG): container finished" podID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerID="5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d" exitCode=2
Sep 17 16:16:07.828667 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:07.828483    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" event={"ID":"8c96b261-795b-40a0-a084-c021bde0b5b7","Type":"ContainerDied","Data":"5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d"}
Sep 17 16:16:08.459680 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:08.459640    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 16:16:08.466196 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:08.466161    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 16:16:12.013829 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.013780    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"]
Sep 17 16:16:12.014346 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.014269    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="storage-initializer"
Sep 17 16:16:12.014346 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.014315    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="storage-initializer"
Sep 17 16:16:12.014346 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.014332    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container"
Sep 17 16:16:12.014346 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.014343    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container"
Sep 17 16:16:12.014557 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.014364    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kube-rbac-proxy"
Sep 17 16:16:12.014557 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.014373    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kube-rbac-proxy"
Sep 17 16:16:12.014557 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.014451    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kserve-container"
Sep 17 16:16:12.014557 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.014465    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5524140a-6e0e-481a-90e1-3ec562e45c62" containerName="kube-rbac-proxy"
Sep 17 16:16:12.017845 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.017826    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.020481 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.020459    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:16:12.020597 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.020465    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 16:16:12.027875 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.027846    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"]
Sep 17 16:16:12.121407 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.121358    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.121600 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.121422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.121600 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.121442    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70011abb-de67-4d9d-86fe-6de2ce225bb4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.121600 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.121563    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.121735 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.121642    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgr2\" (UniqueName: \"kubernetes.io/projected/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-api-access-jqgr2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.222858 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.222827    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.223033 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.222875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgr2\" (UniqueName: \"kubernetes.io/projected/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-api-access-jqgr2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.223033 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:16:12.222999    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 16:16:12.223159 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.223041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.223159 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:16:12.223078    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls podName:70011abb-de67-4d9d-86fe-6de2ce225bb4 nodeName:}" failed. No retries permitted until 2026-09-17 16:16:12.723053915 +0000 UTC m=+1338.988568900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" (UID: "70011abb-de67-4d9d-86fe-6de2ce225bb4") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 16:16:12.223159 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.223146    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.223330 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.223181    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70011abb-de67-4d9d-86fe-6de2ce225bb4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.223605 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.223584    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70011abb-de67-4d9d-86fe-6de2ce225bb4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.223677 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.223657    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.223798 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.223781    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.232717 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.232686    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgr2\" (UniqueName: \"kubernetes.io/projected/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-api-access-jqgr2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.279404 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.279327    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:16:12.425099 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425046    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jszpp\" (UniqueName: \"kubernetes.io/projected/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-api-access-jszpp\") pod \"8c96b261-795b-40a0-a084-c021bde0b5b7\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") "
Sep 17 16:16:12.425099 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425103    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-cabundle-cert\") pod \"8c96b261-795b-40a0-a084-c021bde0b5b7\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") "
Sep 17 16:16:12.425372 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425139    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c96b261-795b-40a0-a084-c021bde0b5b7-proxy-tls\") pod \"8c96b261-795b-40a0-a084-c021bde0b5b7\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") "
Sep 17 16:16:12.425372 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425241    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c96b261-795b-40a0-a084-c021bde0b5b7-kserve-provision-location\") pod \"8c96b261-795b-40a0-a084-c021bde0b5b7\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") "
Sep 17 16:16:12.425463 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425404    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-rbac-proxy-sar-config\") pod \"8c96b261-795b-40a0-a084-c021bde0b5b7\" (UID: \"8c96b261-795b-40a0-a084-c021bde0b5b7\") "
Sep 17 16:16:12.425593 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425557    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c96b261-795b-40a0-a084-c021bde0b5b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c96b261-795b-40a0-a084-c021bde0b5b7" (UID: "8c96b261-795b-40a0-a084-c021bde0b5b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:16:12.425707 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425567    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c96b261-795b-40a0-a084-c021bde0b5b7" (UID: "8c96b261-795b-40a0-a084-c021bde0b5b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:16:12.425707 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425667    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:16:12.425707 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425690    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c96b261-795b-40a0-a084-c021bde0b5b7-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:16:12.425825 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.425750    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c96b261-795b-40a0-a084-c021bde0b5b7" (UID: "8c96b261-795b-40a0-a084-c021bde0b5b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:16:12.427480 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.427464    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c96b261-795b-40a0-a084-c021bde0b5b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c96b261-795b-40a0-a084-c021bde0b5b7" (UID: "8c96b261-795b-40a0-a084-c021bde0b5b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:16:12.427593 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.427579    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-api-access-jszpp" (OuterVolumeSpecName: "kube-api-access-jszpp") pod "8c96b261-795b-40a0-a084-c021bde0b5b7" (UID: "8c96b261-795b-40a0-a084-c021bde0b5b7"). InnerVolumeSpecName "kube-api-access-jszpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:16:12.526388 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.526345    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:16:12.526388 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.526378    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jszpp\" (UniqueName: \"kubernetes.io/projected/8c96b261-795b-40a0-a084-c021bde0b5b7-kube-api-access-jszpp\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:16:12.526388 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.526388    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c96b261-795b-40a0-a084-c021bde0b5b7-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:16:12.727871 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.727781    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.730403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.730384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:12.852239 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.852203    2585 generic.go:356] "Generic (PLEG): container finished" podID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerID="ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c" exitCode=0
Sep 17 16:16:12.852438 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.852265    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"
Sep 17 16:16:12.852438 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.852337    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" event={"ID":"8c96b261-795b-40a0-a084-c021bde0b5b7","Type":"ContainerDied","Data":"ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c"}
Sep 17 16:16:12.852438 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.852401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k" event={"ID":"8c96b261-795b-40a0-a084-c021bde0b5b7","Type":"ContainerDied","Data":"d20adb2c96d46149527436d82234eda1494fae800f0715fc2bc5796b58d706f3"}
Sep 17 16:16:12.852438 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.852429    2585 scope.go:117] "RemoveContainer" containerID="5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d"
Sep 17 16:16:12.861514 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.861498    2585 scope.go:117] "RemoveContainer" containerID="ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c"
Sep 17 16:16:12.869363 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.869342    2585 scope.go:117] "RemoveContainer" containerID="ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c"
Sep 17 16:16:12.874244 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.874215    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"]
Sep 17 16:16:12.877885 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.877868    2585 scope.go:117] "RemoveContainer" containerID="5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d"
Sep 17 16:16:12.878251 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:16:12.878229    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d\": container with ID starting with 5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d not found: ID does not exist" containerID="5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d"
Sep 17 16:16:12.878387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.878261    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d"} err="failed to get container status \"5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d\": rpc error: code = NotFound desc = could not find container \"5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d\": container with ID starting with 5f92cbb87e7187cac53421e3829886829c9bd6cbbd3db422ccdf73e27a97c35d not found: ID does not exist"
Sep 17 16:16:12.878387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.878307    2585 scope.go:117] "RemoveContainer" containerID="ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c"
Sep 17 16:16:12.878781 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:16:12.878754    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c\": container with ID starting with ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c not found: ID does not exist" containerID="ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c"
Sep 17 16:16:12.878877 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.878796    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c"} err="failed to get container status \"ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c\": rpc error: code = NotFound desc = could not find container \"ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c\": container with ID starting with ea66366212715a0130039258e3d434169781899bf49990db4de888c9da69021c not found: ID does not exist"
Sep 17 16:16:12.878877 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.878820    2585 scope.go:117] "RemoveContainer" containerID="ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c"
Sep 17 16:16:12.882366 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:16:12.882334    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c\": container with ID starting with ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c not found: ID does not exist" containerID="ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c"
Sep 17 16:16:12.882487 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.882390    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c"} err="failed to get container status \"ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c\": rpc error: code = NotFound desc = could not find container \"ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c\": container with ID starting with ea99f49f0ccf5e4e8851634e31b2875a19bbb343c94d3d8ecdb3eb1e2533ec7c not found: ID does not exist"
Sep 17 16:16:12.883487 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.883458    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-9cb8k"]
Sep 17 16:16:12.929071 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:12.929024    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:13.206876 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:13.206773    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"]
Sep 17 16:16:13.209380 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:16:13.209350    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70011abb_de67_4d9d_86fe_6de2ce225bb4.slice/crio-4ed93cd99cda5c31aa7d0dd3e876e4304bb77d2ef4c7bedda65f1110b3f0a5c8 WatchSource:0}: Error finding container 4ed93cd99cda5c31aa7d0dd3e876e4304bb77d2ef4c7bedda65f1110b3f0a5c8: Status 404 returned error can't find the container with id 4ed93cd99cda5c31aa7d0dd3e876e4304bb77d2ef4c7bedda65f1110b3f0a5c8
Sep 17 16:16:13.782893 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:13.782845    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" path="/var/lib/kubelet/pods/8c96b261-795b-40a0-a084-c021bde0b5b7/volumes"
Sep 17 16:16:13.859933 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:13.859896    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" event={"ID":"70011abb-de67-4d9d-86fe-6de2ce225bb4","Type":"ContainerStarted","Data":"c23ced9d8f63594c8bfd483c1393d208b6bf7218157c62c800f62c302e4dd4b1"}
Sep 17 16:16:13.860107 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:13.859955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" event={"ID":"70011abb-de67-4d9d-86fe-6de2ce225bb4","Type":"ContainerStarted","Data":"4ed93cd99cda5c31aa7d0dd3e876e4304bb77d2ef4c7bedda65f1110b3f0a5c8"}
Sep 17 16:16:17.878986 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:17.878957    2585 generic.go:356] "Generic (PLEG): container finished" podID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerID="c23ced9d8f63594c8bfd483c1393d208b6bf7218157c62c800f62c302e4dd4b1" exitCode=0
Sep 17 16:16:17.879392 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:17.879018    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" event={"ID":"70011abb-de67-4d9d-86fe-6de2ce225bb4","Type":"ContainerDied","Data":"c23ced9d8f63594c8bfd483c1393d208b6bf7218157c62c800f62c302e4dd4b1"}
Sep 17 16:16:18.885352 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:18.885317    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" event={"ID":"70011abb-de67-4d9d-86fe-6de2ce225bb4","Type":"ContainerStarted","Data":"96109e20e4e1221f070b9e3214d65171f69b5fc99f2b885c2c1d814c0ea7c0a1"}
Sep 17 16:16:18.885826 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:18.885391    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" event={"ID":"70011abb-de67-4d9d-86fe-6de2ce225bb4","Type":"ContainerStarted","Data":"977d8c9431684fd09a6146f833869f80813905aa96bd8bfdbd5447dd070ae097"}
Sep 17 16:16:18.885826 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:18.885597    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:18.885826 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:18.885645    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:18.904740 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:18.904695    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podStartSLOduration=3.9575895279999997 podStartE2EDuration="7.904676287s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.947086759s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:16:18.903943015 +0000 UTC m=+1345.169458015" watchObservedRunningTime="2026-09-17 16:16:18.904676287 +0000 UTC m=+1345.170191291"
Sep 17 16:16:24.896956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:24.896923    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:16:54.897996 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:16:54.897959    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 16:17:04.897753 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:04.897711    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 16:17:14.898348 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:14.898278    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 16:17:24.897818 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:24.897777    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 16:17:35.441736 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:35.441700    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:17:42.148839 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:42.148760    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"]
Sep 17 16:17:42.149337 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:42.149009    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container" containerID="cri-o://977d8c9431684fd09a6146f833869f80813905aa96bd8bfdbd5447dd070ae097" gracePeriod=30
Sep 17 16:17:42.149337 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:42.149042    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kube-rbac-proxy" containerID="cri-o://96109e20e4e1221f070b9e3214d65171f69b5fc99f2b885c2c1d814c0ea7c0a1" gracePeriod=30
Sep 17 16:17:43.245207 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:43.245168    2585 generic.go:356] "Generic (PLEG): container finished" podID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerID="96109e20e4e1221f070b9e3214d65171f69b5fc99f2b885c2c1d814c0ea7c0a1" exitCode=2
Sep 17 16:17:43.245635 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:43.245240    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" event={"ID":"70011abb-de67-4d9d-86fe-6de2ce225bb4","Type":"ContainerDied","Data":"96109e20e4e1221f070b9e3214d65171f69b5fc99f2b885c2c1d814c0ea7c0a1"}
Sep 17 16:17:44.891046 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:44.890997    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 17 16:17:44.897557 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:44.897524    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 16:17:48.271047 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.271017    2585 generic.go:356] "Generic (PLEG): container finished" podID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerID="977d8c9431684fd09a6146f833869f80813905aa96bd8bfdbd5447dd070ae097" exitCode=0
Sep 17 16:17:48.271465 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.271085    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" event={"ID":"70011abb-de67-4d9d-86fe-6de2ce225bb4","Type":"ContainerDied","Data":"977d8c9431684fd09a6146f833869f80813905aa96bd8bfdbd5447dd070ae097"}
Sep 17 16:17:48.690921 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.690894    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:17:48.787485 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.787445    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-rbac-proxy-sar-config\") pod \"70011abb-de67-4d9d-86fe-6de2ce225bb4\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") "
Sep 17 16:17:48.787485 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.787493    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-cabundle-cert\") pod \"70011abb-de67-4d9d-86fe-6de2ce225bb4\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") "
Sep 17 16:17:48.787734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.787553    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70011abb-de67-4d9d-86fe-6de2ce225bb4-kserve-provision-location\") pod \"70011abb-de67-4d9d-86fe-6de2ce225bb4\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") "
Sep 17 16:17:48.787734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.787577    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls\") pod \"70011abb-de67-4d9d-86fe-6de2ce225bb4\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") "
Sep 17 16:17:48.787734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.787658    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqgr2\" (UniqueName: \"kubernetes.io/projected/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-api-access-jqgr2\") pod \"70011abb-de67-4d9d-86fe-6de2ce225bb4\" (UID: \"70011abb-de67-4d9d-86fe-6de2ce225bb4\") "
Sep 17 16:17:48.787964 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.787940    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70011abb-de67-4d9d-86fe-6de2ce225bb4" (UID: "70011abb-de67-4d9d-86fe-6de2ce225bb4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:48.787964 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.787952    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70011abb-de67-4d9d-86fe-6de2ce225bb4" (UID: "70011abb-de67-4d9d-86fe-6de2ce225bb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:48.788058 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.787959    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70011abb-de67-4d9d-86fe-6de2ce225bb4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70011abb-de67-4d9d-86fe-6de2ce225bb4" (UID: "70011abb-de67-4d9d-86fe-6de2ce225bb4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:17:48.789998 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.789968    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-api-access-jqgr2" (OuterVolumeSpecName: "kube-api-access-jqgr2") pod "70011abb-de67-4d9d-86fe-6de2ce225bb4" (UID: "70011abb-de67-4d9d-86fe-6de2ce225bb4"). InnerVolumeSpecName "kube-api-access-jqgr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:48.790096 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.790072    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70011abb-de67-4d9d-86fe-6de2ce225bb4" (UID: "70011abb-de67-4d9d-86fe-6de2ce225bb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:48.888751 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.888645    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:17:48.888751 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.888694    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70011abb-de67-4d9d-86fe-6de2ce225bb4-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:17:48.888751 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.888705    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70011abb-de67-4d9d-86fe-6de2ce225bb4-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:17:48.888751 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.888716    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70011abb-de67-4d9d-86fe-6de2ce225bb4-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:17:48.888751 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:48.888727    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgr2\" (UniqueName: \"kubernetes.io/projected/70011abb-de67-4d9d-86fe-6de2ce225bb4-kube-api-access-jqgr2\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:17:49.278115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:49.278085    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"
Sep 17 16:17:49.278504 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:49.278091    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r" event={"ID":"70011abb-de67-4d9d-86fe-6de2ce225bb4","Type":"ContainerDied","Data":"4ed93cd99cda5c31aa7d0dd3e876e4304bb77d2ef4c7bedda65f1110b3f0a5c8"}
Sep 17 16:17:49.278504 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:49.278234    2585 scope.go:117] "RemoveContainer" containerID="96109e20e4e1221f070b9e3214d65171f69b5fc99f2b885c2c1d814c0ea7c0a1"
Sep 17 16:17:49.287669 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:49.287649    2585 scope.go:117] "RemoveContainer" containerID="977d8c9431684fd09a6146f833869f80813905aa96bd8bfdbd5447dd070ae097"
Sep 17 16:17:49.295662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:49.295640    2585 scope.go:117] "RemoveContainer" containerID="c23ced9d8f63594c8bfd483c1393d208b6bf7218157c62c800f62c302e4dd4b1"
Sep 17 16:17:49.307299 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:49.307259    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"]
Sep 17 16:17:49.312673 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:49.312653    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-l7t5r"]
Sep 17 16:17:49.782899 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:17:49.782863    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" path="/var/lib/kubelet/pods/70011abb-de67-4d9d-86fe-6de2ce225bb4/volumes"
Sep 17 16:18:04.919840 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.919799    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-ab81d85c"]
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920114    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="storage-initializer"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920125    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="storage-initializer"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920133    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kube-rbac-proxy"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920139    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kube-rbac-proxy"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920146    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="storage-initializer"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920151    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="storage-initializer"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920166    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kube-rbac-proxy"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920171    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kube-rbac-proxy"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920181    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920186    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920196    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920200    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920244    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kube-rbac-proxy"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920252    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c96b261-795b-40a0-a084-c021bde0b5b7" containerName="kserve-container"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920259    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kserve-container"
Sep 17 16:18:04.920331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.920266    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="70011abb-de67-4d9d-86fe-6de2ce225bb4" containerName="kube-rbac-proxy"
Sep 17 16:18:04.924758 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.924737    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ab81d85c"
Sep 17 16:18:04.927322 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.927275    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8hq\" (UniqueName: \"kubernetes.io/projected/be7bd8b6-fa36-484e-833e-97244abdf478-kube-api-access-zq8hq\") pod \"curl-metrics-ab81d85c\" (UID: \"be7bd8b6-fa36-484e-833e-97244abdf478\") " pod="kserve/curl-metrics-ab81d85c"
Sep 17 16:18:04.927576 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.927555    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:18:04.927660 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.927582    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:18:04.927660 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.927557    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9gvx7\""
Sep 17 16:18:04.931182 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:04.931155    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-ab81d85c"]
Sep 17 16:18:05.028052 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:05.028014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8hq\" (UniqueName: \"kubernetes.io/projected/be7bd8b6-fa36-484e-833e-97244abdf478-kube-api-access-zq8hq\") pod \"curl-metrics-ab81d85c\" (UID: \"be7bd8b6-fa36-484e-833e-97244abdf478\") " pod="kserve/curl-metrics-ab81d85c"
Sep 17 16:18:05.039966 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:05.039932    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8hq\" (UniqueName: \"kubernetes.io/projected/be7bd8b6-fa36-484e-833e-97244abdf478-kube-api-access-zq8hq\") pod \"curl-metrics-ab81d85c\" (UID: \"be7bd8b6-fa36-484e-833e-97244abdf478\") " pod="kserve/curl-metrics-ab81d85c"
Sep 17 16:18:05.234072 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:05.233977    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ab81d85c"
Sep 17 16:18:05.514246 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:05.514208    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-ab81d85c"]
Sep 17 16:18:05.518707 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:18:05.518676    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe7bd8b6_fa36_484e_833e_97244abdf478.slice/crio-b94e7b605a8a1d82bcd04acd4735b6659c54e5c0577cd6a4454cb244c4052aa6 WatchSource:0}: Error finding container b94e7b605a8a1d82bcd04acd4735b6659c54e5c0577cd6a4454cb244c4052aa6: Status 404 returned error can't find the container with id b94e7b605a8a1d82bcd04acd4735b6659c54e5c0577cd6a4454cb244c4052aa6
Sep 17 16:18:05.521176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:05.521157    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:18:06.309917 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:18:06.309878    2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe7bd8b6_fa36_484e_833e_97244abdf478.slice/crio-conmon-8af1a79f97bdecd364cb1c712fddfbd54a342f66d273c1a97ea40bb0508d3711.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe7bd8b6_fa36_484e_833e_97244abdf478.slice/crio-8af1a79f97bdecd364cb1c712fddfbd54a342f66d273c1a97ea40bb0508d3711.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 16:18:06.352122 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:06.352087    2585 generic.go:356] "Generic (PLEG): container finished" podID="be7bd8b6-fa36-484e-833e-97244abdf478" containerID="8af1a79f97bdecd364cb1c712fddfbd54a342f66d273c1a97ea40bb0508d3711" exitCode=0
Sep 17 16:18:06.352318 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:06.352172    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-ab81d85c" event={"ID":"be7bd8b6-fa36-484e-833e-97244abdf478","Type":"ContainerDied","Data":"8af1a79f97bdecd364cb1c712fddfbd54a342f66d273c1a97ea40bb0508d3711"}
Sep 17 16:18:06.352318 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:06.352215    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-ab81d85c" event={"ID":"be7bd8b6-fa36-484e-833e-97244abdf478","Type":"ContainerStarted","Data":"b94e7b605a8a1d82bcd04acd4735b6659c54e5c0577cd6a4454cb244c4052aa6"}
Sep 17 16:18:07.489654 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:07.489625    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ab81d85c"
Sep 17 16:18:07.545125 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:07.545088    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq8hq\" (UniqueName: \"kubernetes.io/projected/be7bd8b6-fa36-484e-833e-97244abdf478-kube-api-access-zq8hq\") pod \"be7bd8b6-fa36-484e-833e-97244abdf478\" (UID: \"be7bd8b6-fa36-484e-833e-97244abdf478\") "
Sep 17 16:18:07.547404 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:07.547370    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7bd8b6-fa36-484e-833e-97244abdf478-kube-api-access-zq8hq" (OuterVolumeSpecName: "kube-api-access-zq8hq") pod "be7bd8b6-fa36-484e-833e-97244abdf478" (UID: "be7bd8b6-fa36-484e-833e-97244abdf478"). InnerVolumeSpecName "kube-api-access-zq8hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:07.551660 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:07.551631    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-ab81d85c_be7bd8b6-fa36-484e-833e-97244abdf478/curl-metrics-ab81d85c/0.log"
Sep 17 16:18:07.587122 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:07.587090    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-ab81d85c"]
Sep 17 16:18:07.590763 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:07.590738    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-ab81d85c"]
Sep 17 16:18:07.646584 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:07.646506    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8hq\" (UniqueName: \"kubernetes.io/projected/be7bd8b6-fa36-484e-833e-97244abdf478-kube-api-access-zq8hq\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:18:07.786883 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:07.786844    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7bd8b6-fa36-484e-833e-97244abdf478" path="/var/lib/kubelet/pods/be7bd8b6-fa36-484e-833e-97244abdf478/volumes"
Sep 17 16:18:08.362054 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:08.361997    2585 scope.go:117] "RemoveContainer" containerID="8af1a79f97bdecd364cb1c712fddfbd54a342f66d273c1a97ea40bb0508d3711"
Sep 17 16:18:08.362248 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:08.362030    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ab81d85c"
Sep 17 16:18:13.027135 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.027099    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-b7eecdfb"]
Sep 17 16:18:13.027631 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.027434    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7bd8b6-fa36-484e-833e-97244abdf478" containerName="curl-metrics-ab81d85c"
Sep 17 16:18:13.027631 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.027445    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7bd8b6-fa36-484e-833e-97244abdf478" containerName="curl-metrics-ab81d85c"
Sep 17 16:18:13.027631 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.027505    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7bd8b6-fa36-484e-833e-97244abdf478" containerName="curl-metrics-ab81d85c"
Sep 17 16:18:13.031784 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.031765    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b7eecdfb"
Sep 17 16:18:13.040597 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.040571    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:18:13.041581 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.041562    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:18:13.041707 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.041564    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9gvx7\""
Sep 17 16:18:13.042723 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.042704    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-b7eecdfb"]
Sep 17 16:18:13.090627 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.090587    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6fcl\" (UniqueName: \"kubernetes.io/projected/cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b-kube-api-access-j6fcl\") pod \"curl-metrics-b7eecdfb\" (UID: \"cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b\") " pod="kserve/curl-metrics-b7eecdfb"
Sep 17 16:18:13.191916 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.191874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6fcl\" (UniqueName: \"kubernetes.io/projected/cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b-kube-api-access-j6fcl\") pod \"curl-metrics-b7eecdfb\" (UID: \"cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b\") " pod="kserve/curl-metrics-b7eecdfb"
Sep 17 16:18:13.206496 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.206465    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6fcl\" (UniqueName: \"kubernetes.io/projected/cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b-kube-api-access-j6fcl\") pod \"curl-metrics-b7eecdfb\" (UID: \"cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b\") " pod="kserve/curl-metrics-b7eecdfb"
Sep 17 16:18:13.342346 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.342207    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b7eecdfb"
Sep 17 16:18:13.624469 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:13.624242    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-b7eecdfb"]
Sep 17 16:18:13.627193 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:18:13.627168    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf6ad0d0_d9dd_4bfe_b122_8df0fc14b58b.slice/crio-2fcdfb09f65cba165d6df551040330b27dc50d6f9d09e2bf3ca9e82844750a1e WatchSource:0}: Error finding container 2fcdfb09f65cba165d6df551040330b27dc50d6f9d09e2bf3ca9e82844750a1e: Status 404 returned error can't find the container with id 2fcdfb09f65cba165d6df551040330b27dc50d6f9d09e2bf3ca9e82844750a1e
Sep 17 16:18:14.392364 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:14.392329    2585 generic.go:356] "Generic (PLEG): container finished" podID="cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b" containerID="fee80ea8e78e010b5b1df9598265edb0ff95002c56952f854fe7596a0adbfd4d" exitCode=0
Sep 17 16:18:14.392782 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:14.392413    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-b7eecdfb" event={"ID":"cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b","Type":"ContainerDied","Data":"fee80ea8e78e010b5b1df9598265edb0ff95002c56952f854fe7596a0adbfd4d"}
Sep 17 16:18:14.392782 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:14.392455    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-b7eecdfb" event={"ID":"cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b","Type":"ContainerStarted","Data":"2fcdfb09f65cba165d6df551040330b27dc50d6f9d09e2bf3ca9e82844750a1e"}
Sep 17 16:18:15.526755 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:15.526728    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b7eecdfb"
Sep 17 16:18:15.563648 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:15.563621    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-b7eecdfb_cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b/curl-metrics-b7eecdfb/0.log"
Sep 17 16:18:15.598155 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:15.598115    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-b7eecdfb"]
Sep 17 16:18:15.605122 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:15.605096    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-b7eecdfb"]
Sep 17 16:18:15.613812 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:15.613778    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6fcl\" (UniqueName: \"kubernetes.io/projected/cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b-kube-api-access-j6fcl\") pod \"cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b\" (UID: \"cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b\") "
Sep 17 16:18:15.616050 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:15.616020    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b-kube-api-access-j6fcl" (OuterVolumeSpecName: "kube-api-access-j6fcl") pod "cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b" (UID: "cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b"). InnerVolumeSpecName "kube-api-access-j6fcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:15.715141 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:15.715048    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6fcl\" (UniqueName: \"kubernetes.io/projected/cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b-kube-api-access-j6fcl\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:18:15.782128 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:15.782084    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b" path="/var/lib/kubelet/pods/cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b/volumes"
Sep 17 16:18:16.402364 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:16.402331    2585 scope.go:117] "RemoveContainer" containerID="fee80ea8e78e010b5b1df9598265edb0ff95002c56952f854fe7596a0adbfd4d"
Sep 17 16:18:16.402544 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:16.402341    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b7eecdfb"
Sep 17 16:18:21.147649 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.147613    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"]
Sep 17 16:18:21.148075 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.147935    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b" containerName="curl-metrics-b7eecdfb"
Sep 17 16:18:21.148075 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.147947    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b" containerName="curl-metrics-b7eecdfb"
Sep 17 16:18:21.148075 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.148014    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6ad0d0-d9dd-4bfe-b122-8df0fc14b58b" containerName="curl-metrics-b7eecdfb"
Sep 17 16:18:21.152683 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.152658    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.156254 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.156235    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 16:18:21.157123 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.157103    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:18:21.157602 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.157581    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 16:18:21.157913 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.157890    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:18:21.158062 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.158028    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:18:21.158062 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.158034    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:18:21.158196 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.158177    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:18:21.176301 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.176262    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"]
Sep 17 16:18:21.267533 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.267501    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-cabundle-cert\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.267533 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.267537    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.267768 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.267605    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9dvx\" (UniqueName: \"kubernetes.io/projected/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-api-access-h9dvx\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.267768 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.267641    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.267768 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.267657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.368883 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.368830    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9dvx\" (UniqueName: \"kubernetes.io/projected/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-api-access-h9dvx\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.369093 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.368906    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.369093 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.368932    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.369093 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.368976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-cabundle-cert\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.369093 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.369000    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.369093 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:18:21.369053    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 17 16:18:21.369411 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:18:21.369124    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls podName:a1c105d6-9fbf-4587-afa8-88ba1ca167f8 nodeName:}" failed. No retries permitted until 2026-09-17 16:18:21.869103962 +0000 UTC m=+1468.134618954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls") pod "isvc-sklearn-predictor-5f789bcb8f-dnmxq" (UID: "a1c105d6-9fbf-4587-afa8-88ba1ca167f8") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 17 16:18:21.369534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.369515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.369710 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.369692    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.369875 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.369856    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-cabundle-cert\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.383131 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.383098    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9dvx\" (UniqueName: \"kubernetes.io/projected/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-api-access-h9dvx\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.873334 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.873258    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:21.875917 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:21.875885    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls\") pod \"isvc-sklearn-predictor-5f789bcb8f-dnmxq\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:22.063982 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:22.063929    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:22.348782 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:22.348733    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"]
Sep 17 16:18:22.351613 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:18:22.351576    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c105d6_9fbf_4587_afa8_88ba1ca167f8.slice/crio-a7a42a5ce59a74957224db0dc8583ad879dc86d5b4f6be68fc41a082fc3554d9 WatchSource:0}: Error finding container a7a42a5ce59a74957224db0dc8583ad879dc86d5b4f6be68fc41a082fc3554d9: Status 404 returned error can't find the container with id a7a42a5ce59a74957224db0dc8583ad879dc86d5b4f6be68fc41a082fc3554d9
Sep 17 16:18:22.431976 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:22.431937    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" event={"ID":"a1c105d6-9fbf-4587-afa8-88ba1ca167f8","Type":"ContainerStarted","Data":"9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9"}
Sep 17 16:18:22.432145 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:22.432005    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" event={"ID":"a1c105d6-9fbf-4587-afa8-88ba1ca167f8","Type":"ContainerStarted","Data":"a7a42a5ce59a74957224db0dc8583ad879dc86d5b4f6be68fc41a082fc3554d9"}
Sep 17 16:18:26.451088 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:26.450988    2585 generic.go:356] "Generic (PLEG): container finished" podID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerID="9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9" exitCode=0
Sep 17 16:18:26.451088 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:26.451071    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" event={"ID":"a1c105d6-9fbf-4587-afa8-88ba1ca167f8","Type":"ContainerDied","Data":"9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9"}
Sep 17 16:18:27.458842 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:27.458811    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" event={"ID":"a1c105d6-9fbf-4587-afa8-88ba1ca167f8","Type":"ContainerStarted","Data":"3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd"}
Sep 17 16:18:27.459260 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:27.458870    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" event={"ID":"a1c105d6-9fbf-4587-afa8-88ba1ca167f8","Type":"ContainerStarted","Data":"c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396"}
Sep 17 16:18:27.459260 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:27.459142    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:27.459409 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:27.459276    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:27.460618 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:27.460582    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:18:27.507614 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:27.507551    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podStartSLOduration=2.774314611 podStartE2EDuration="6.50753446s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.733219849s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:27.506468833 +0000 UTC m=+1473.771983845" watchObservedRunningTime="2026-09-17 16:18:27.50753446 +0000 UTC m=+1473.773049459"
Sep 17 16:18:28.463919 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:28.463876    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:18:33.468461 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:33.468432    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:18:33.469109 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:33.469007    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:18:43.469105 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:43.469063    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:18:53.469564 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:53.469522    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:18:55.823129 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:55.823095    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:18:55.824189 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:18:55.824169    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:19:03.469770 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:03.469723    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:19:13.469759 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:13.469672    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:19:23.469413 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:23.469368    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:19:33.468832 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:33.468789    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:19:41.782816 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:41.782782    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:19:51.085628 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:51.085582    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"]
Sep 17 16:19:51.086176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:51.085824    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" containerID="cri-o://c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396" gracePeriod=30
Sep 17 16:19:51.086176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:51.085861    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kube-rbac-proxy" containerID="cri-o://3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd" gracePeriod=30
Sep 17 16:19:51.778534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:51.778484    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:19:51.825388 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:51.825354    2585 generic.go:356] "Generic (PLEG): container finished" podID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerID="3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd" exitCode=2
Sep 17 16:19:51.825546 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:51.825428    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" event={"ID":"a1c105d6-9fbf-4587-afa8-88ba1ca167f8","Type":"ContainerDied","Data":"3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd"}
Sep 17 16:19:53.464574 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:53.464531    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 16:19:55.734403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.734381    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:19:55.807235 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807175    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-rbac-proxy-sar-config\") pod \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") "
Sep 17 16:19:55.807235 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807232    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kserve-provision-location\") pod \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") "
Sep 17 16:19:55.807407 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807346    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls\") pod \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") "
Sep 17 16:19:55.807407 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807395    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-cabundle-cert\") pod \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") "
Sep 17 16:19:55.807478 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807462    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9dvx\" (UniqueName: \"kubernetes.io/projected/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-api-access-h9dvx\") pod \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\" (UID: \"a1c105d6-9fbf-4587-afa8-88ba1ca167f8\") "
Sep 17 16:19:55.807578 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807544    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1c105d6-9fbf-4587-afa8-88ba1ca167f8" (UID: "a1c105d6-9fbf-4587-afa8-88ba1ca167f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:55.807703 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807575    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1c105d6-9fbf-4587-afa8-88ba1ca167f8" (UID: "a1c105d6-9fbf-4587-afa8-88ba1ca167f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:55.807773 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807764    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:19:55.807829 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807776    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:19:55.807829 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.807783    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1c105d6-9fbf-4587-afa8-88ba1ca167f8" (UID: "a1c105d6-9fbf-4587-afa8-88ba1ca167f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:55.809461 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.809440    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1c105d6-9fbf-4587-afa8-88ba1ca167f8" (UID: "a1c105d6-9fbf-4587-afa8-88ba1ca167f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:19:55.809558 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.809532    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-api-access-h9dvx" (OuterVolumeSpecName: "kube-api-access-h9dvx") pod "a1c105d6-9fbf-4587-afa8-88ba1ca167f8" (UID: "a1c105d6-9fbf-4587-afa8-88ba1ca167f8"). InnerVolumeSpecName "kube-api-access-h9dvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:19:55.844115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.844089    2585 generic.go:356] "Generic (PLEG): container finished" podID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerID="c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396" exitCode=0
Sep 17 16:19:55.844202 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.844144    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"
Sep 17 16:19:55.844202 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.844176    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" event={"ID":"a1c105d6-9fbf-4587-afa8-88ba1ca167f8","Type":"ContainerDied","Data":"c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396"}
Sep 17 16:19:55.844274 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.844243    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq" event={"ID":"a1c105d6-9fbf-4587-afa8-88ba1ca167f8","Type":"ContainerDied","Data":"a7a42a5ce59a74957224db0dc8583ad879dc86d5b4f6be68fc41a082fc3554d9"}
Sep 17 16:19:55.844348 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.844280    2585 scope.go:117] "RemoveContainer" containerID="3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd"
Sep 17 16:19:55.852974 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.852956    2585 scope.go:117] "RemoveContainer" containerID="c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396"
Sep 17 16:19:55.860500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.860480    2585 scope.go:117] "RemoveContainer" containerID="9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9"
Sep 17 16:19:55.866012 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.865987    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"]
Sep 17 16:19:55.868345 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.868327    2585 scope.go:117] "RemoveContainer" containerID="3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd"
Sep 17 16:19:55.868764 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:19:55.868738    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd\": container with ID starting with 3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd not found: ID does not exist" containerID="3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd"
Sep 17 16:19:55.868869 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.868776    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd"} err="failed to get container status \"3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd\": rpc error: code = NotFound desc = could not find container \"3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd\": container with ID starting with 3474d6c6052ffe7f37a15464444b2fffa41f5a03ec8286204688b9d15cbf38fd not found: ID does not exist"
Sep 17 16:19:55.868869 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.868799    2585 scope.go:117] "RemoveContainer" containerID="c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396"
Sep 17 16:19:55.869143 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:19:55.869121    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396\": container with ID starting with c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396 not found: ID does not exist" containerID="c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396"
Sep 17 16:19:55.869198 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.869153    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396"} err="failed to get container status \"c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396\": rpc error: code = NotFound desc = could not find container \"c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396\": container with ID starting with c5aa59abf01d97c78092056714f5b74533492b2fcd3c97d54d07f2c341a02396 not found: ID does not exist"
Sep 17 16:19:55.869198 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.869170    2585 scope.go:117] "RemoveContainer" containerID="9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9"
Sep 17 16:19:55.869528 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:19:55.869502    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9\": container with ID starting with 9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9 not found: ID does not exist" containerID="9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9"
Sep 17 16:19:55.869605 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.869527    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9"} err="failed to get container status \"9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9\": rpc error: code = NotFound desc = could not find container \"9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9\": container with ID starting with 9c4f5aadae16ec2b88ba94abd6fa6ae0a490b308959c2fe1b34b4d2c158382e9 not found: ID does not exist"
Sep 17 16:19:55.870273 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.870255    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f789bcb8f-dnmxq"]
Sep 17 16:19:55.909029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.909005    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9dvx\" (UniqueName: \"kubernetes.io/projected/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-kube-api-access-h9dvx\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:19:55.909029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.909026    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:19:55.909149 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:55.909038    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1c105d6-9fbf-4587-afa8-88ba1ca167f8-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:19:57.782152 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:19:57.782122    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" path="/var/lib/kubelet/pods/a1c105d6-9fbf-4587-afa8-88ba1ca167f8/volumes"
Sep 17 16:22:12.957875 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.957782    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"]
Sep 17 16:22:12.958327 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.958177    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kube-rbac-proxy"
Sep 17 16:22:12.958327 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.958195    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kube-rbac-proxy"
Sep 17 16:22:12.958327 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.958218    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="storage-initializer"
Sep 17 16:22:12.958327 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.958224    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="storage-initializer"
Sep 17 16:22:12.958327 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.958231    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container"
Sep 17 16:22:12.958327 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.958237    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container"
Sep 17 16:22:12.958327 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.958311    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kserve-container"
Sep 17 16:22:12.958327 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.958322    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c105d6-9fbf-4587-afa8-88ba1ca167f8" containerName="kube-rbac-proxy"
Sep 17 16:22:12.961618 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.961600    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:12.964532 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.964509    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 16:22:12.964696 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.964587    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:22:12.965683 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.965659    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:22:12.965834 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.965738    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:22:12.965834 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.965741    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:22:12.965834 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.965815    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 16:22:12.965986 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.965858    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:22:12.993471 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:12.993431    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"]
Sep 17 16:22:13.102902 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.102862    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.102902 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.102916    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b583870d-ed67-494b-a4ed-4d7fddb90c32-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.103151 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.102958    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b583870d-ed67-494b-a4ed-4d7fddb90c32-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.103151 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.103005    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmgtl\" (UniqueName: \"kubernetes.io/projected/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-api-access-hmgtl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.103151 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.103039    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.204303 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.204259    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b583870d-ed67-494b-a4ed-4d7fddb90c32-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.204522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.204336    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b583870d-ed67-494b-a4ed-4d7fddb90c32-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.204522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.204418    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmgtl\" (UniqueName: \"kubernetes.io/projected/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-api-access-hmgtl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.204522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.204470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.204522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.204500    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.204844 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.204818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b583870d-ed67-494b-a4ed-4d7fddb90c32-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.205117 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.205094    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.205117 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.205105    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.207086 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.207064    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b583870d-ed67-494b-a4ed-4d7fddb90c32-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.213818 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.213753    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmgtl\" (UniqueName: \"kubernetes.io/projected/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-api-access-hmgtl\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.271648 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.271612    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:22:13.556453 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:13.556425    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"]
Sep 17 16:22:13.558642 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:22:13.558615    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb583870d_ed67_494b_a4ed_4d7fddb90c32.slice/crio-cad665b0a81f3e7b26e25dc936917f57556c5a2a8603c7ee198e8bcff735a96b WatchSource:0}: Error finding container cad665b0a81f3e7b26e25dc936917f57556c5a2a8603c7ee198e8bcff735a96b: Status 404 returned error can't find the container with id cad665b0a81f3e7b26e25dc936917f57556c5a2a8603c7ee198e8bcff735a96b
Sep 17 16:22:14.433173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:14.433133    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" event={"ID":"b583870d-ed67-494b-a4ed-4d7fddb90c32","Type":"ContainerStarted","Data":"28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d"}
Sep 17 16:22:14.433579 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:14.433193    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" event={"ID":"b583870d-ed67-494b-a4ed-4d7fddb90c32","Type":"ContainerStarted","Data":"cad665b0a81f3e7b26e25dc936917f57556c5a2a8603c7ee198e8bcff735a96b"}
Sep 17 16:22:22.466361 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:22.466316    2585 generic.go:356] "Generic (PLEG): container finished" podID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerID="28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d" exitCode=0
Sep 17 16:22:22.466802 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:22:22.466370    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" event={"ID":"b583870d-ed67-494b-a4ed-4d7fddb90c32","Type":"ContainerDied","Data":"28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d"}
Sep 17 16:23:11.875207 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.875172    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"]
Sep 17 16:23:11.878464 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.878437    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:11.882052 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.882025    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 16:23:11.887081 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.887058    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:23:11.913339 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.913301    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"]
Sep 17 16:23:11.952817 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.952783    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnld\" (UniqueName: \"kubernetes.io/projected/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-api-access-4dnld\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:11.953013 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.952822    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:11.953013 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.952864    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:11.953013 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.952893    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:11.953013 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:11.952930    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.054043 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.054004    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnld\" (UniqueName: \"kubernetes.io/projected/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-api-access-4dnld\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.054043 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.054056    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.054335 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.054098    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.054504 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.054469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.054606 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.054555    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.054728 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:23:12.054708    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 17 16:23:12.054802 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:23:12.054781    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls podName:d4b81f49-1b02-4648-a321-6eaa11a9cf17 nodeName:}" failed. No retries permitted until 2026-09-17 16:23:12.554761368 +0000 UTC m=+1758.820276347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls") pod "isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" (UID: "d4b81f49-1b02-4648-a321-6eaa11a9cf17") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 17 16:23:12.054954 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.054904    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.054954 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.054945    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.055130 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.054955    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.066956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.066921    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnld\" (UniqueName: \"kubernetes.io/projected/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-api-access-4dnld\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.558803 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.558765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.561605 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.561569    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6777c5d8bf-46czv\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:12.792713 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:12.792676    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:13.085432 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:13.085391    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"]
Sep 17 16:23:13.089572 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:23:13.089537    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b81f49_1b02_4648_a321_6eaa11a9cf17.slice/crio-7bd69b9c39a14c427ed58aeb90ed74b6baad2a95e949cbe5048e125d17d988e1 WatchSource:0}: Error finding container 7bd69b9c39a14c427ed58aeb90ed74b6baad2a95e949cbe5048e125d17d988e1: Status 404 returned error can't find the container with id 7bd69b9c39a14c427ed58aeb90ed74b6baad2a95e949cbe5048e125d17d988e1
Sep 17 16:23:13.727584 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:13.727545    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" event={"ID":"d4b81f49-1b02-4648-a321-6eaa11a9cf17","Type":"ContainerStarted","Data":"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91"}
Sep 17 16:23:13.727785 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:13.727623    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" event={"ID":"d4b81f49-1b02-4648-a321-6eaa11a9cf17","Type":"ContainerStarted","Data":"7bd69b9c39a14c427ed58aeb90ed74b6baad2a95e949cbe5048e125d17d988e1"}
Sep 17 16:23:19.763818 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:19.763778    2585 generic.go:356] "Generic (PLEG): container finished" podID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerID="83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91" exitCode=0
Sep 17 16:23:19.764339 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:19.763834    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" event={"ID":"d4b81f49-1b02-4648-a321-6eaa11a9cf17","Type":"ContainerDied","Data":"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91"}
Sep 17 16:23:20.772842 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:20.772802    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" event={"ID":"d4b81f49-1b02-4648-a321-6eaa11a9cf17","Type":"ContainerStarted","Data":"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404"}
Sep 17 16:23:20.773407 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:20.772875    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" event={"ID":"d4b81f49-1b02-4648-a321-6eaa11a9cf17","Type":"ContainerStarted","Data":"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6"}
Sep 17 16:23:20.773407 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:20.773208    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:20.796964 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:20.796907    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podStartSLOduration=4.005746206 podStartE2EDuration="9.796888264s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.791142058s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:23:20.795127853 +0000 UTC m=+1767.060642853" watchObservedRunningTime="2026-09-17 16:23:20.796888264 +0000 UTC m=+1767.062403264"
Sep 17 16:23:21.779706 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:21.779669    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:23:21.783822 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:21.783794    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:22.781780 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:22.781734    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:23:27.786717 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:27.786688    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:27.787280 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:27.787251    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:23:37.787506 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:37.787413    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:23:47.787758 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:47.787729    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:23:53.962730 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:53.962688    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"]
Sep 17 16:23:53.963449 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:53.962970    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container" containerID="cri-o://eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6" gracePeriod=30
Sep 17 16:23:53.963449 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:53.963005    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy" containerID="cri-o://03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404" gracePeriod=30
Sep 17 16:23:57.782499 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:57.782454    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:23:57.787976 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:23:57.787923    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:24:02.782765 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:02.782721    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:24:07.781983 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:07.781945    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:24:07.785015 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:07.784991    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:24:07.787499 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:07.787467    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:24:12.782777 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:12.782718    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:24:17.782361 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:17.782316    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:24:17.787687 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:17.787658    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 16:24:17.787823 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:17.787765    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:24:22.782404 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:22.782356    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:24:25.565302 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.565263    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:24:25.693405 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.693276    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls\") pod \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") "
Sep 17 16:24:25.693405 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.693367    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dnld\" (UniqueName: \"kubernetes.io/projected/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-api-access-4dnld\") pod \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") "
Sep 17 16:24:25.693405 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.693406    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-cabundle-cert\") pod \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") "
Sep 17 16:24:25.694273 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.693485    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-rbac-proxy-sar-config\") pod \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") "
Sep 17 16:24:25.694273 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.693532    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kserve-provision-location\") pod \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\" (UID: \"d4b81f49-1b02-4648-a321-6eaa11a9cf17\") "
Sep 17 16:24:25.694273 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.694175    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4b81f49-1b02-4648-a321-6eaa11a9cf17" (UID: "d4b81f49-1b02-4648-a321-6eaa11a9cf17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:24:25.694273 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.694193    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4b81f49-1b02-4648-a321-6eaa11a9cf17" (UID: "d4b81f49-1b02-4648-a321-6eaa11a9cf17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:24:25.696237 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.696209    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4b81f49-1b02-4648-a321-6eaa11a9cf17" (UID: "d4b81f49-1b02-4648-a321-6eaa11a9cf17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:24:25.696587 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.696553    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-api-access-4dnld" (OuterVolumeSpecName: "kube-api-access-4dnld") pod "d4b81f49-1b02-4648-a321-6eaa11a9cf17" (UID: "d4b81f49-1b02-4648-a321-6eaa11a9cf17"). InnerVolumeSpecName "kube-api-access-4dnld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:24:25.714959 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.714919    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4b81f49-1b02-4648-a321-6eaa11a9cf17" (UID: "d4b81f49-1b02-4648-a321-6eaa11a9cf17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:25.794617 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.794582    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4b81f49-1b02-4648-a321-6eaa11a9cf17-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:24:25.794617 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.794613    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnld\" (UniqueName: \"kubernetes.io/projected/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-api-access-4dnld\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:24:25.794617 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.794623    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:24:25.794850 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.794632    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:24:25.794850 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.794642    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b81f49-1b02-4648-a321-6eaa11a9cf17-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:24:25.868724 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.868696    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:24:25.868724 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:25.868713    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:24:26.117640 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.117604    2585 generic.go:356] "Generic (PLEG): container finished" podID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerID="03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404" exitCode=2
Sep 17 16:24:26.117640 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.117633    2585 generic.go:356] "Generic (PLEG): container finished" podID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerID="eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6" exitCode=0
Sep 17 16:24:26.117932 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.117694    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"
Sep 17 16:24:26.117932 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.117692    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" event={"ID":"d4b81f49-1b02-4648-a321-6eaa11a9cf17","Type":"ContainerDied","Data":"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404"}
Sep 17 16:24:26.117932 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.117860    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" event={"ID":"d4b81f49-1b02-4648-a321-6eaa11a9cf17","Type":"ContainerDied","Data":"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6"}
Sep 17 16:24:26.117932 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.117884    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv" event={"ID":"d4b81f49-1b02-4648-a321-6eaa11a9cf17","Type":"ContainerDied","Data":"7bd69b9c39a14c427ed58aeb90ed74b6baad2a95e949cbe5048e125d17d988e1"}
Sep 17 16:24:26.117932 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.117909    2585 scope.go:117] "RemoveContainer" containerID="03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404"
Sep 17 16:24:26.127316 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.127256    2585 scope.go:117] "RemoveContainer" containerID="eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6"
Sep 17 16:24:26.135266 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.135244    2585 scope.go:117] "RemoveContainer" containerID="83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91"
Sep 17 16:24:26.137943 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.137918    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"]
Sep 17 16:24:26.144378 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.144346    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6777c5d8bf-46czv"]
Sep 17 16:24:26.145687 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.145665    2585 scope.go:117] "RemoveContainer" containerID="03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404"
Sep 17 16:24:26.146037 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:24:26.146015    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404\": container with ID starting with 03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404 not found: ID does not exist" containerID="03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404"
Sep 17 16:24:26.146115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.146047    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404"} err="failed to get container status \"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404\": rpc error: code = NotFound desc = could not find container \"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404\": container with ID starting with 03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404 not found: ID does not exist"
Sep 17 16:24:26.146115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.146068    2585 scope.go:117] "RemoveContainer" containerID="eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6"
Sep 17 16:24:26.146400 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:24:26.146375    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6\": container with ID starting with eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6 not found: ID does not exist" containerID="eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6"
Sep 17 16:24:26.146500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.146407    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6"} err="failed to get container status \"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6\": rpc error: code = NotFound desc = could not find container \"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6\": container with ID starting with eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6 not found: ID does not exist"
Sep 17 16:24:26.146500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.146424    2585 scope.go:117] "RemoveContainer" containerID="83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91"
Sep 17 16:24:26.146734 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:24:26.146716    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91\": container with ID starting with 83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91 not found: ID does not exist" containerID="83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91"
Sep 17 16:24:26.146804 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.146742    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91"} err="failed to get container status \"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91\": rpc error: code = NotFound desc = could not find container \"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91\": container with ID starting with 83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91 not found: ID does not exist"
Sep 17 16:24:26.146804 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.146763    2585 scope.go:117] "RemoveContainer" containerID="03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404"
Sep 17 16:24:26.147036 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.147018    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404"} err="failed to get container status \"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404\": rpc error: code = NotFound desc = could not find container \"03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404\": container with ID starting with 03897d69481dda4df1dc191c2e263f79912b4b7b213f719a80581a265b2e7404 not found: ID does not exist"
Sep 17 16:24:26.147099 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.147037    2585 scope.go:117] "RemoveContainer" containerID="eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6"
Sep 17 16:24:26.147329 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.147278    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6"} err="failed to get container status \"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6\": rpc error: code = NotFound desc = could not find container \"eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6\": container with ID starting with eed7eb5bd06823903faf3bab3672970f9d417268e2cd117e6dd9491478ae7ab6 not found: ID does not exist"
Sep 17 16:24:26.147423 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.147332    2585 scope.go:117] "RemoveContainer" containerID="83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91"
Sep 17 16:24:26.147604 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:26.147583    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91"} err="failed to get container status \"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91\": rpc error: code = NotFound desc = could not find container \"83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91\": container with ID starting with 83941879a7f796fcfd39c168259efc9e01fbcb732a343c9a7fa28f9af884ea91 not found: ID does not exist"
Sep 17 16:24:27.126157 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:27.126122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" event={"ID":"b583870d-ed67-494b-a4ed-4d7fddb90c32","Type":"ContainerStarted","Data":"65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56"}
Sep 17 16:24:27.126749 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:27.126183    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" event={"ID":"b583870d-ed67-494b-a4ed-4d7fddb90c32","Type":"ContainerStarted","Data":"1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25"}
Sep 17 16:24:27.126749 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:27.126408    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:24:27.126749 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:27.126551    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:24:27.127705 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:27.127675    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:24:27.153981 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:27.153923    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" podStartSLOduration=3.1615533989999998 podStartE2EDuration="2m15.153906456s" totalImagesPullingTime="2m3.989303974s" totalInitContainerRuntime="8.003049083s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:24:27.153656793 +0000 UTC m=+1833.419171797" watchObservedRunningTime="2026-09-17 16:24:27.153906456 +0000 UTC m=+1833.419421456"
Sep 17 16:24:27.782120 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:27.782078    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" path="/var/lib/kubelet/pods/d4b81f49-1b02-4648-a321-6eaa11a9cf17/volumes"
Sep 17 16:24:28.130694 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:28.130588    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:24:33.135697 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:33.135665    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:24:33.136176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:33.136156    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:24:43.136463 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:24:43.136434    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:25:04.385300 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385243    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"]
Sep 17 16:25:04.385799 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385585    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container"
Sep 17 16:25:04.385799 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385597    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container"
Sep 17 16:25:04.385799 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385611    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="storage-initializer"
Sep 17 16:25:04.385799 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385616    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="storage-initializer"
Sep 17 16:25:04.385799 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385633    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy"
Sep 17 16:25:04.385799 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385640    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy"
Sep 17 16:25:04.385799 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385694    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kserve-container"
Sep 17 16:25:04.385799 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.385701    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b81f49-1b02-4648-a321-6eaa11a9cf17" containerName="kube-rbac-proxy"
Sep 17 16:25:04.390146 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.390124    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.392755 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.392729    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 16:25:04.392904 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.392738    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:25:04.399176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.399149    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"]
Sep 17 16:25:04.434595 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.434557    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbpg\" (UniqueName: \"kubernetes.io/projected/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-api-access-xnbpg\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.434772 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.434606    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.434772 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.434634    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-proxy-tls\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.434772 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.434724    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.434877 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.434768    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.535999 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.535953    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbpg\" (UniqueName: \"kubernetes.io/projected/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-api-access-xnbpg\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.536202 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.536028    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.536202 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.536077    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-proxy-tls\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.536202 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.536117    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.536202 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.536142    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.536648 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.536624    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.536882 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.536851    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.537010 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.536931    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.538920 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.538891    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-proxy-tls\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.545213 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.545182    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbpg\" (UniqueName: \"kubernetes.io/projected/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-api-access-xnbpg\") pod \"isvc-sklearn-v2-predictor-855f948db9-s9skj\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.703368 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.703251    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:04.982495 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:04.982339    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"]
Sep 17 16:25:04.984939 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:25:04.984908    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc017d51_5e8b_4b47_aa0f_ac444b0c7083.slice/crio-301c7279a1980c086d809fc33ce8502ad6b7cf9b9d230657e406bb8fefcab509 WatchSource:0}: Error finding container 301c7279a1980c086d809fc33ce8502ad6b7cf9b9d230657e406bb8fefcab509: Status 404 returned error can't find the container with id 301c7279a1980c086d809fc33ce8502ad6b7cf9b9d230657e406bb8fefcab509
Sep 17 16:25:05.304305 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:05.304238    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" event={"ID":"fc017d51-5e8b-4b47-aa0f-ac444b0c7083","Type":"ContainerStarted","Data":"916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb"}
Sep 17 16:25:05.304491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:05.304355    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" event={"ID":"fc017d51-5e8b-4b47-aa0f-ac444b0c7083","Type":"ContainerStarted","Data":"301c7279a1980c086d809fc33ce8502ad6b7cf9b9d230657e406bb8fefcab509"}
Sep 17 16:25:06.338178 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:06.338142    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"]
Sep 17 16:25:06.338603 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:06.338404    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kserve-container" containerID="cri-o://1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25" gracePeriod=30
Sep 17 16:25:06.338603 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:06.338445    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kube-rbac-proxy" containerID="cri-o://65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56" gracePeriod=30
Sep 17 16:25:07.317699 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:07.317658    2585 generic.go:356] "Generic (PLEG): container finished" podID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerID="65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56" exitCode=2
Sep 17 16:25:07.317882 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:07.317730    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" event={"ID":"b583870d-ed67-494b-a4ed-4d7fddb90c32","Type":"ContainerDied","Data":"65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56"}
Sep 17 16:25:08.131487 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.131441    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 16:25:08.281637 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.281612    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:25:08.324911 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.324880    2585 generic.go:356] "Generic (PLEG): container finished" podID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerID="1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25" exitCode=0
Sep 17 16:25:08.325088 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.324940    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"
Sep 17 16:25:08.325088 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.324969    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" event={"ID":"b583870d-ed67-494b-a4ed-4d7fddb90c32","Type":"ContainerDied","Data":"1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25"}
Sep 17 16:25:08.325088 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.325060    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt" event={"ID":"b583870d-ed67-494b-a4ed-4d7fddb90c32","Type":"ContainerDied","Data":"cad665b0a81f3e7b26e25dc936917f57556c5a2a8603c7ee198e8bcff735a96b"}
Sep 17 16:25:08.325210 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.325093    2585 scope.go:117] "RemoveContainer" containerID="65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56"
Sep 17 16:25:08.333644 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.333624    2585 scope.go:117] "RemoveContainer" containerID="1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25"
Sep 17 16:25:08.341557 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.341533    2585 scope.go:117] "RemoveContainer" containerID="28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d"
Sep 17 16:25:08.349393 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.349372    2585 scope.go:117] "RemoveContainer" containerID="65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56"
Sep 17 16:25:08.349725 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:25:08.349701    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56\": container with ID starting with 65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56 not found: ID does not exist" containerID="65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56"
Sep 17 16:25:08.349819 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.349740    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56"} err="failed to get container status \"65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56\": rpc error: code = NotFound desc = could not find container \"65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56\": container with ID starting with 65c4e28fbee89293868030522375d90323dea721a89c7711355a9562e641dd56 not found: ID does not exist"
Sep 17 16:25:08.349819 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.349760    2585 scope.go:117] "RemoveContainer" containerID="1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25"
Sep 17 16:25:08.350069 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:25:08.350051    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25\": container with ID starting with 1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25 not found: ID does not exist" containerID="1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25"
Sep 17 16:25:08.350114 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.350075    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25"} err="failed to get container status \"1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25\": rpc error: code = NotFound desc = could not find container \"1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25\": container with ID starting with 1f8f6995def7385c251ffcacf0beaaefe48b8e8abd43de8db805d337d4871f25 not found: ID does not exist"
Sep 17 16:25:08.350114 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.350092    2585 scope.go:117] "RemoveContainer" containerID="28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d"
Sep 17 16:25:08.350363 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:25:08.350345    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d\": container with ID starting with 28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d not found: ID does not exist" containerID="28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d"
Sep 17 16:25:08.350427 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.350370    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d"} err="failed to get container status \"28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d\": rpc error: code = NotFound desc = could not find container \"28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d\": container with ID starting with 28691c9717ca9600b5df2fa9b4d66fed49fbf34139f88c4c0558aad14118180d not found: ID does not exist"
Sep 17 16:25:08.371419 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.371385    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b583870d-ed67-494b-a4ed-4d7fddb90c32-proxy-tls\") pod \"b583870d-ed67-494b-a4ed-4d7fddb90c32\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") "
Sep 17 16:25:08.371419 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.371430    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-cabundle-cert\") pod \"b583870d-ed67-494b-a4ed-4d7fddb90c32\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") "
Sep 17 16:25:08.371642 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.371457    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b583870d-ed67-494b-a4ed-4d7fddb90c32-kserve-provision-location\") pod \"b583870d-ed67-494b-a4ed-4d7fddb90c32\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") "
Sep 17 16:25:08.371642 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.371496    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-rbac-proxy-sar-config\") pod \"b583870d-ed67-494b-a4ed-4d7fddb90c32\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") "
Sep 17 16:25:08.371642 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.371571    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmgtl\" (UniqueName: \"kubernetes.io/projected/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-api-access-hmgtl\") pod \"b583870d-ed67-494b-a4ed-4d7fddb90c32\" (UID: \"b583870d-ed67-494b-a4ed-4d7fddb90c32\") "
Sep 17 16:25:08.371908 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.371873    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b583870d-ed67-494b-a4ed-4d7fddb90c32" (UID: "b583870d-ed67-494b-a4ed-4d7fddb90c32"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:08.371986 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.371909    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b583870d-ed67-494b-a4ed-4d7fddb90c32" (UID: "b583870d-ed67-494b-a4ed-4d7fddb90c32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:08.372478 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.372459    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b583870d-ed67-494b-a4ed-4d7fddb90c32-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b583870d-ed67-494b-a4ed-4d7fddb90c32" (UID: "b583870d-ed67-494b-a4ed-4d7fddb90c32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:08.373739 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.373722    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b583870d-ed67-494b-a4ed-4d7fddb90c32-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b583870d-ed67-494b-a4ed-4d7fddb90c32" (UID: "b583870d-ed67-494b-a4ed-4d7fddb90c32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:25:08.373966 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.373943    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-api-access-hmgtl" (OuterVolumeSpecName: "kube-api-access-hmgtl") pod "b583870d-ed67-494b-a4ed-4d7fddb90c32" (UID: "b583870d-ed67-494b-a4ed-4d7fddb90c32"). InnerVolumeSpecName "kube-api-access-hmgtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:25:08.472749 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.472709    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:25:08.472749 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.472742    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmgtl\" (UniqueName: \"kubernetes.io/projected/b583870d-ed67-494b-a4ed-4d7fddb90c32-kube-api-access-hmgtl\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:25:08.472950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.472752    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b583870d-ed67-494b-a4ed-4d7fddb90c32-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:25:08.472950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.472774    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b583870d-ed67-494b-a4ed-4d7fddb90c32-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:25:08.472950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.472782    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b583870d-ed67-494b-a4ed-4d7fddb90c32-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:25:08.651973 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.651937    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"]
Sep 17 16:25:08.654383 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:08.654354    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-sl4vt"]
Sep 17 16:25:09.331254 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:09.331215    2585 generic.go:356] "Generic (PLEG): container finished" podID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerID="916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb" exitCode=0
Sep 17 16:25:09.331746 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:09.331276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" event={"ID":"fc017d51-5e8b-4b47-aa0f-ac444b0c7083","Type":"ContainerDied","Data":"916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb"}
Sep 17 16:25:09.782471 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:09.782441    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" path="/var/lib/kubelet/pods/b583870d-ed67-494b-a4ed-4d7fddb90c32/volumes"
Sep 17 16:25:10.337638 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:10.337602    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" event={"ID":"fc017d51-5e8b-4b47-aa0f-ac444b0c7083","Type":"ContainerStarted","Data":"d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551"}
Sep 17 16:25:10.338143 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:10.337669    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" event={"ID":"fc017d51-5e8b-4b47-aa0f-ac444b0c7083","Type":"ContainerStarted","Data":"98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249"}
Sep 17 16:25:10.338143 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:10.337870    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:10.361738 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:10.361680    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podStartSLOduration=2.321958118 podStartE2EDuration="6.361665241s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.039707123s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:10.358900702 +0000 UTC m=+1876.624415701" watchObservedRunningTime="2026-09-17 16:25:10.361665241 +0000 UTC m=+1876.627180240"
Sep 17 16:25:11.342223 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:11.342186    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:11.343485 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:11.343457    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:25:12.346541 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:12.346501    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:25:17.352166 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:17.352137    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:25:17.352638 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:17.352614    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:25:27.353111 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:27.353067    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:25:37.353012 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:37.352971    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:25:47.352722 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:47.352676    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:25:53.204729 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.204695    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"]
Sep 17 16:25:53.205132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.205022    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kserve-container"
Sep 17 16:25:53.205132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.205033    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kserve-container"
Sep 17 16:25:53.205132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.205046    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="storage-initializer"
Sep 17 16:25:53.205132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.205052    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="storage-initializer"
Sep 17 16:25:53.205132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.205062    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kube-rbac-proxy"
Sep 17 16:25:53.205132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.205067    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kube-rbac-proxy"
Sep 17 16:25:53.205132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.205120    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kserve-container"
Sep 17 16:25:53.205132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.205131    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b583870d-ed67-494b-a4ed-4d7fddb90c32" containerName="kube-rbac-proxy"
Sep 17 16:25:53.208369 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.208337    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.225131 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.225105    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 16:25:53.225131 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.225120    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 16:25:53.259518 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.259481    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"]
Sep 17 16:25:53.278654 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.278617    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3cdb019-77bf-49c0-8e87-152aab55b231-proxy-tls\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.278850 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.278670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wssm\" (UniqueName: \"kubernetes.io/projected/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-api-access-8wssm\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.278850 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.278699    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.278850 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.278781    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.278850 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.278821    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3cdb019-77bf-49c0-8e87-152aab55b231-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.338552 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.338514    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"]
Sep 17 16:25:53.342244 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.342224    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.347879 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.347853    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 16:25:53.371265 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.371234    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"]
Sep 17 16:25:53.379535 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379497    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3cdb019-77bf-49c0-8e87-152aab55b231-proxy-tls\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.379708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379542    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wssm\" (UniqueName: \"kubernetes.io/projected/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-api-access-8wssm\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.379708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379563    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.379708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379587    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.379708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379632    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.379708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379648    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctndw\" (UniqueName: \"kubernetes.io/projected/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-api-access-ctndw\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.379708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.380061 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379711    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.380061 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379740    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.380061 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.379774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3cdb019-77bf-49c0-8e87-152aab55b231-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.380400 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.380368    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.380503 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.380444    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.380503 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.380479    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3cdb019-77bf-49c0-8e87-152aab55b231-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.382246 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.382225    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3cdb019-77bf-49c0-8e87-152aab55b231-proxy-tls\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.397357 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.397327    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wssm\" (UniqueName: \"kubernetes.io/projected/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-api-access-8wssm\") pod \"isvc-canary-create-v2-predictor-cdb97449-nglgz\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.481000 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.480890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.481000 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.480952    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.481000 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.480969    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctndw\" (UniqueName: \"kubernetes.io/projected/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-api-access-ctndw\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.481000 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.481001    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.481398 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.481022    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.481623 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.481594    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.481764 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.481745    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.481816 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.481799    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.483710 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.483683    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.507842 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.507807    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctndw\" (UniqueName: \"kubernetes.io/projected/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-api-access-ctndw\") pod \"isvc-canary-create-predictor-66d85d8bfb-4bnsp\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.520471 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.520443    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:53.653222 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.653175    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:53.805473 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.805274    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"]
Sep 17 16:25:53.808162 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:25:53.808130    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cdb019_77bf_49c0_8e87_152aab55b231.slice/crio-c12e3362f67040348aa529496657264069bd8d637135f025bfdc0a9dc35e4d2e WatchSource:0}: Error finding container c12e3362f67040348aa529496657264069bd8d637135f025bfdc0a9dc35e4d2e: Status 404 returned error can't find the container with id c12e3362f67040348aa529496657264069bd8d637135f025bfdc0a9dc35e4d2e
Sep 17 16:25:53.931949 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:53.931925    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"]
Sep 17 16:25:53.934117 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:25:53.934089    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5562595b_3732_4b02_bb1a_d71f0bc2a3a6.slice/crio-2c7d37a3bf1cd6f68bdf22fb089699aab3df8e64043ab97ee3c38f05d90820bc WatchSource:0}: Error finding container 2c7d37a3bf1cd6f68bdf22fb089699aab3df8e64043ab97ee3c38f05d90820bc: Status 404 returned error can't find the container with id 2c7d37a3bf1cd6f68bdf22fb089699aab3df8e64043ab97ee3c38f05d90820bc
Sep 17 16:25:54.529687 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:54.529643    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" event={"ID":"5562595b-3732-4b02-bb1a-d71f0bc2a3a6","Type":"ContainerStarted","Data":"2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b"}
Sep 17 16:25:54.530202 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:54.529736    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" event={"ID":"5562595b-3732-4b02-bb1a-d71f0bc2a3a6","Type":"ContainerStarted","Data":"2c7d37a3bf1cd6f68bdf22fb089699aab3df8e64043ab97ee3c38f05d90820bc"}
Sep 17 16:25:54.531418 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:54.531385    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" event={"ID":"b3cdb019-77bf-49c0-8e87-152aab55b231","Type":"ContainerStarted","Data":"626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327"}
Sep 17 16:25:54.531557 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:54.531452    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" event={"ID":"b3cdb019-77bf-49c0-8e87-152aab55b231","Type":"ContainerStarted","Data":"c12e3362f67040348aa529496657264069bd8d637135f025bfdc0a9dc35e4d2e"}
Sep 17 16:25:57.352785 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:57.352746    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:25:58.554988 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:58.554949    2585 generic.go:356] "Generic (PLEG): container finished" podID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerID="2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b" exitCode=0
Sep 17 16:25:58.555465 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:58.555023    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" event={"ID":"5562595b-3732-4b02-bb1a-d71f0bc2a3a6","Type":"ContainerDied","Data":"2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b"}
Sep 17 16:25:58.556752 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:58.556727    2585 generic.go:356] "Generic (PLEG): container finished" podID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerID="626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327" exitCode=0
Sep 17 16:25:58.556855 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:58.556797    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" event={"ID":"b3cdb019-77bf-49c0-8e87-152aab55b231","Type":"ContainerDied","Data":"626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327"}
Sep 17 16:25:59.564426 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.564390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" event={"ID":"5562595b-3732-4b02-bb1a-d71f0bc2a3a6","Type":"ContainerStarted","Data":"7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662"}
Sep 17 16:25:59.564893 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.564451    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" event={"ID":"5562595b-3732-4b02-bb1a-d71f0bc2a3a6","Type":"ContainerStarted","Data":"55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f"}
Sep 17 16:25:59.564893 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.564800    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:59.564893 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.564862    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:25:59.566641 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.566603    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:25:59.566810 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.566789    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" event={"ID":"b3cdb019-77bf-49c0-8e87-152aab55b231","Type":"ContainerStarted","Data":"f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d"}
Sep 17 16:25:59.566866 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.566841    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" event={"ID":"b3cdb019-77bf-49c0-8e87-152aab55b231","Type":"ContainerStarted","Data":"e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0"}
Sep 17 16:25:59.567005 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.566994    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:25:59.587892 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.587827    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podStartSLOduration=2.580864121 podStartE2EDuration="6.587805699s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.006941578s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:59.587052248 +0000 UTC m=+1925.852567248" watchObservedRunningTime="2026-09-17 16:25:59.587805699 +0000 UTC m=+1925.853320697"
Sep 17 16:25:59.614999 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:25:59.614943    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podStartSLOduration=2.564956918 podStartE2EDuration="6.614925343s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.049968425s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:59.614099938 +0000 UTC m=+1925.879614937" watchObservedRunningTime="2026-09-17 16:25:59.614925343 +0000 UTC m=+1925.880440345"
Sep 17 16:26:00.571536 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:00.571488    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:26:00.572056 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:00.571512    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:26:00.572734 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:00.572708    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:26:01.577353 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:01.577313    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:26:05.575853 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:05.575824    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:26:05.576357 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:05.576330    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:26:06.582546 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:06.582518    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:26:06.583016 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:06.582960    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:26:07.352782 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:07.352739    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:26:15.577378 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:15.577337    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:26:16.583792 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:16.583750    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:26:17.353158 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:17.353119    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:26:25.576668 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:25.576625    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:26:26.583892 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:26.583850    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:26:27.353406 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:27.353371    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:26:34.580960 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:34.580924    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"]
Sep 17 16:26:34.581402 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:34.581158    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" containerID="cri-o://98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249" gracePeriod=30
Sep 17 16:26:34.581402 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:34.581198    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kube-rbac-proxy" containerID="cri-o://d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551" gracePeriod=30
Sep 17 16:26:34.745326 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:34.745270    2585 generic.go:356] "Generic (PLEG): container finished" podID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerID="d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551" exitCode=2
Sep 17 16:26:34.745502 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:34.745350    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" event={"ID":"fc017d51-5e8b-4b47-aa0f-ac444b0c7083","Type":"ContainerDied","Data":"d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551"}
Sep 17 16:26:35.576496 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:35.576462    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:26:36.583818 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:36.583731    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:26:37.347215 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:37.347170    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 16:26:37.353240 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:37.353214    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:26:39.624475 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.624453    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:26:39.708634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.708551    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnbpg\" (UniqueName: \"kubernetes.io/projected/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-api-access-xnbpg\") pod \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") "
Sep 17 16:26:39.708634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.708590    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kserve-provision-location\") pod \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") "
Sep 17 16:26:39.708634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.708608    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-cabundle-cert\") pod \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") "
Sep 17 16:26:39.708634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.708635    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-rbac-proxy-sar-config\") pod \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") "
Sep 17 16:26:39.708969 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.708675    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-proxy-tls\") pod \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\" (UID: \"fc017d51-5e8b-4b47-aa0f-ac444b0c7083\") "
Sep 17 16:26:39.709028 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.708929    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc017d51-5e8b-4b47-aa0f-ac444b0c7083" (UID: "fc017d51-5e8b-4b47-aa0f-ac444b0c7083"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:26:39.709081 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.709036    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc017d51-5e8b-4b47-aa0f-ac444b0c7083" (UID: "fc017d51-5e8b-4b47-aa0f-ac444b0c7083"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:26:39.709081 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.709058    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc017d51-5e8b-4b47-aa0f-ac444b0c7083" (UID: "fc017d51-5e8b-4b47-aa0f-ac444b0c7083"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:26:39.710830 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.710804    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-api-access-xnbpg" (OuterVolumeSpecName: "kube-api-access-xnbpg") pod "fc017d51-5e8b-4b47-aa0f-ac444b0c7083" (UID: "fc017d51-5e8b-4b47-aa0f-ac444b0c7083"). InnerVolumeSpecName "kube-api-access-xnbpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:26:39.710931 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.710860    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc017d51-5e8b-4b47-aa0f-ac444b0c7083" (UID: "fc017d51-5e8b-4b47-aa0f-ac444b0c7083"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:26:39.764096 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764063    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"]
Sep 17 16:26:39.764446 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764433    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kube-rbac-proxy"
Sep 17 16:26:39.764500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764448    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kube-rbac-proxy"
Sep 17 16:26:39.764500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764460    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container"
Sep 17 16:26:39.764500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764466    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container"
Sep 17 16:26:39.764500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764473    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="storage-initializer"
Sep 17 16:26:39.764500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764479    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="storage-initializer"
Sep 17 16:26:39.764704 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764545    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kube-rbac-proxy"
Sep 17 16:26:39.764704 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.764557    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerName="kserve-container"
Sep 17 16:26:39.767901 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.767885    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.770399 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.770351    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 16:26:39.770399 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.770380    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 16:26:39.772046 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.772017    2585 generic.go:356] "Generic (PLEG): container finished" podID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" containerID="98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249" exitCode=0
Sep 17 16:26:39.772156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.772101    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"
Sep 17 16:26:39.772156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.772111    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" event={"ID":"fc017d51-5e8b-4b47-aa0f-ac444b0c7083","Type":"ContainerDied","Data":"98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249"}
Sep 17 16:26:39.772309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.772181    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj" event={"ID":"fc017d51-5e8b-4b47-aa0f-ac444b0c7083","Type":"ContainerDied","Data":"301c7279a1980c086d809fc33ce8502ad6b7cf9b9d230657e406bb8fefcab509"}
Sep 17 16:26:39.772309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.772220    2585 scope.go:117] "RemoveContainer" containerID="d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551"
Sep 17 16:26:39.783792 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.783773    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"]
Sep 17 16:26:39.786918 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.786895    2585 scope.go:117] "RemoveContainer" containerID="98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249"
Sep 17 16:26:39.796216 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.796200    2585 scope.go:117] "RemoveContainer" containerID="916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb"
Sep 17 16:26:39.805480 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.805461    2585 scope.go:117] "RemoveContainer" containerID="d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551"
Sep 17 16:26:39.805754 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:26:39.805728    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551\": container with ID starting with d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551 not found: ID does not exist" containerID="d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551"
Sep 17 16:26:39.805838 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.805763    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551"} err="failed to get container status \"d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551\": rpc error: code = NotFound desc = could not find container \"d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551\": container with ID starting with d286218f99f2bd5faa4346c9ceb884304b5b27a0520183dbb383c69f8baa9551 not found: ID does not exist"
Sep 17 16:26:39.805838 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.805782    2585 scope.go:117] "RemoveContainer" containerID="98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249"
Sep 17 16:26:39.806073 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:26:39.806056    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249\": container with ID starting with 98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249 not found: ID does not exist" containerID="98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249"
Sep 17 16:26:39.806116 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.806079    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249"} err="failed to get container status \"98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249\": rpc error: code = NotFound desc = could not find container \"98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249\": container with ID starting with 98359ed912c10a0df6237796dc024af5e208ef39e953ced63ee80e71b5fd6249 not found: ID does not exist"
Sep 17 16:26:39.806116 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.806094    2585 scope.go:117] "RemoveContainer" containerID="916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb"
Sep 17 16:26:39.806351 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:26:39.806334    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb\": container with ID starting with 916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb not found: ID does not exist" containerID="916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb"
Sep 17 16:26:39.806396 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.806355    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb"} err="failed to get container status \"916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb\": rpc error: code = NotFound desc = could not find container \"916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb\": container with ID starting with 916224aa933823fa26ac68c0a8afdecf59f395b32554016c59704143a4d7d9cb not found: ID does not exist"
Sep 17 16:26:39.809743 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.809725    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e2e5405-058a-4a65-ba74-2356783a3e76-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.809808 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.809757    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.809808 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.809778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2e5405-058a-4a65-ba74-2356783a3e76-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.809917 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.809883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.809971 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.809919    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgxdt\" (UniqueName: \"kubernetes.io/projected/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-api-access-sgxdt\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.810045 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.810031    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbpg\" (UniqueName: \"kubernetes.io/projected/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-api-access-xnbpg\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:26:39.810106 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.810050    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:26:39.810106 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.810067    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:26:39.810106 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.810081    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:26:39.810106 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.810094    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc017d51-5e8b-4b47-aa0f-ac444b0c7083-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:26:39.813654 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.813633    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"]
Sep 17 16:26:39.819482 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.819460    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-855f948db9-s9skj"]
Sep 17 16:26:39.910881 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.910849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e2e5405-058a-4a65-ba74-2356783a3e76-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.911053 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.910892    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.911053 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.910914    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2e5405-058a-4a65-ba74-2356783a3e76-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.911053 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.910956    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.911053 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.910981    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxdt\" (UniqueName: \"kubernetes.io/projected/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-api-access-sgxdt\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.911425 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.911399    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2e5405-058a-4a65-ba74-2356783a3e76-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.911625 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.911601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.911693 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.911655    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.913629 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.913613    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e2e5405-058a-4a65-ba74-2356783a3e76-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:39.919819 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:39.919799    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgxdt\" (UniqueName: \"kubernetes.io/projected/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-api-access-sgxdt\") pod \"isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:40.081155 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:40.081119    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:40.359156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:40.359126    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"]
Sep 17 16:26:40.361168 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:26:40.361138    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2e5405_058a_4a65_ba74_2356783a3e76.slice/crio-8961752eefaa4d6cf16b71e288de7d2575ce3959a07c288f8db88752d92e496e WatchSource:0}: Error finding container 8961752eefaa4d6cf16b71e288de7d2575ce3959a07c288f8db88752d92e496e: Status 404 returned error can't find the container with id 8961752eefaa4d6cf16b71e288de7d2575ce3959a07c288f8db88752d92e496e
Sep 17 16:26:40.779045 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:40.779005    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" event={"ID":"2e2e5405-058a-4a65-ba74-2356783a3e76","Type":"ContainerStarted","Data":"a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360"}
Sep 17 16:26:40.779534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:40.779084    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" event={"ID":"2e2e5405-058a-4a65-ba74-2356783a3e76","Type":"ContainerStarted","Data":"8961752eefaa4d6cf16b71e288de7d2575ce3959a07c288f8db88752d92e496e"}
Sep 17 16:26:41.783128 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:41.783091    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc017d51-5e8b-4b47-aa0f-ac444b0c7083" path="/var/lib/kubelet/pods/fc017d51-5e8b-4b47-aa0f-ac444b0c7083/volumes"
Sep 17 16:26:44.805622 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:44.805586    2585 generic.go:356] "Generic (PLEG): container finished" podID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerID="a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360" exitCode=0
Sep 17 16:26:44.806014 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:44.805664    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" event={"ID":"2e2e5405-058a-4a65-ba74-2356783a3e76","Type":"ContainerDied","Data":"a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360"}
Sep 17 16:26:45.576275 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:45.576236    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:26:45.812215 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:45.812171    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" event={"ID":"2e2e5405-058a-4a65-ba74-2356783a3e76","Type":"ContainerStarted","Data":"38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094"}
Sep 17 16:26:45.812633 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:45.812237    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" event={"ID":"2e2e5405-058a-4a65-ba74-2356783a3e76","Type":"ContainerStarted","Data":"5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19"}
Sep 17 16:26:45.812633 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:45.812454    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:45.846420 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:45.846269    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podStartSLOduration=2.8631665699999997 podStartE2EDuration="6.846244233s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.983077663s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:26:45.845147016 +0000 UTC m=+1972.110662018" watchObservedRunningTime="2026-09-17 16:26:45.846244233 +0000 UTC m=+1972.111759234"
Sep 17 16:26:46.583983 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:46.583941    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:26:46.817571 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:46.817536    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:46.818854 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:46.818822    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:26:47.822725 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:47.822674    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:26:52.828881 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:52.828836    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:26:52.829477 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:52.829449    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:26:55.577220 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:55.577176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:26:56.583779 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:26:56.583735    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:27:02.829797 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:02.829760    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:27:05.576307 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:05.576235    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:27:06.583662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:06.583620    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:27:12.777630 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:12.777584    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:27:12.830027 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:12.829980    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:27:16.583500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:16.583470    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:27:16.729057 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:16.729024    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"]
Sep 17 16:27:16.729256 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:16.729235    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container" containerID="cri-o://55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f" gracePeriod=30
Sep 17 16:27:16.729346 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:16.729315    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kube-rbac-proxy" containerID="cri-o://7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662" gracePeriod=30
Sep 17 16:27:16.730262 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:16.730243    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:27:16.965465 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:16.965430    2585 generic.go:356] "Generic (PLEG): container finished" podID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerID="7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662" exitCode=2
Sep 17 16:27:16.965624 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:16.965505    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" event={"ID":"5562595b-3732-4b02-bb1a-d71f0bc2a3a6","Type":"ContainerDied","Data":"7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662"}
Sep 17 16:27:20.571809 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:20.571764    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 16:27:21.175522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.175495    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:27:21.279874 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.279846    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-proxy-tls\") pod \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") "
Sep 17 16:27:21.280029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.279907    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctndw\" (UniqueName: \"kubernetes.io/projected/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-api-access-ctndw\") pod \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") "
Sep 17 16:27:21.280029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.279934    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kserve-provision-location\") pod \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") "
Sep 17 16:27:21.280111 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.280030    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-cabundle-cert\") pod \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") "
Sep 17 16:27:21.280111 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.280073    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-rbac-proxy-sar-config\") pod \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\" (UID: \"5562595b-3732-4b02-bb1a-d71f0bc2a3a6\") "
Sep 17 16:27:21.280345 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.280318    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5562595b-3732-4b02-bb1a-d71f0bc2a3a6" (UID: "5562595b-3732-4b02-bb1a-d71f0bc2a3a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:21.280476 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.280413    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5562595b-3732-4b02-bb1a-d71f0bc2a3a6" (UID: "5562595b-3732-4b02-bb1a-d71f0bc2a3a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:21.280524 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.280486    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5562595b-3732-4b02-bb1a-d71f0bc2a3a6" (UID: "5562595b-3732-4b02-bb1a-d71f0bc2a3a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:21.282234 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.282212    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5562595b-3732-4b02-bb1a-d71f0bc2a3a6" (UID: "5562595b-3732-4b02-bb1a-d71f0bc2a3a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:27:21.282234 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.282222    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-api-access-ctndw" (OuterVolumeSpecName: "kube-api-access-ctndw") pod "5562595b-3732-4b02-bb1a-d71f0bc2a3a6" (UID: "5562595b-3732-4b02-bb1a-d71f0bc2a3a6"). InnerVolumeSpecName "kube-api-access-ctndw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:27:21.381379 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.381346    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:21.381379 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.381372    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:21.381379 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.381383    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:21.381584 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.381392    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctndw\" (UniqueName: \"kubernetes.io/projected/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kube-api-access-ctndw\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:21.381584 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.381401    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5562595b-3732-4b02-bb1a-d71f0bc2a3a6-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:21.990425 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.990392    2585 generic.go:356] "Generic (PLEG): container finished" podID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerID="55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f" exitCode=0
Sep 17 16:27:21.990907 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.990459    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"
Sep 17 16:27:21.990907 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.990478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" event={"ID":"5562595b-3732-4b02-bb1a-d71f0bc2a3a6","Type":"ContainerDied","Data":"55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f"}
Sep 17 16:27:21.990907 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.990548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp" event={"ID":"5562595b-3732-4b02-bb1a-d71f0bc2a3a6","Type":"ContainerDied","Data":"2c7d37a3bf1cd6f68bdf22fb089699aab3df8e64043ab97ee3c38f05d90820bc"}
Sep 17 16:27:21.990907 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.990575    2585 scope.go:117] "RemoveContainer" containerID="7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662"
Sep 17 16:27:21.999297 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:21.999263    2585 scope.go:117] "RemoveContainer" containerID="55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f"
Sep 17 16:27:22.006922 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.006907    2585 scope.go:117] "RemoveContainer" containerID="2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b"
Sep 17 16:27:22.011574 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.011553    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"]
Sep 17 16:27:22.015139 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.015123    2585 scope.go:117] "RemoveContainer" containerID="7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662"
Sep 17 16:27:22.015457 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:27:22.015436    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662\": container with ID starting with 7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662 not found: ID does not exist" containerID="7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662"
Sep 17 16:27:22.015530 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.015464    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662"} err="failed to get container status \"7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662\": rpc error: code = NotFound desc = could not find container \"7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662\": container with ID starting with 7362eea6aa2ccbfe16518d2c25fc894e38415009ce828a57ad0749a502503662 not found: ID does not exist"
Sep 17 16:27:22.015530 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.015490    2585 scope.go:117] "RemoveContainer" containerID="55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f"
Sep 17 16:27:22.015794 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:27:22.015774    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f\": container with ID starting with 55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f not found: ID does not exist" containerID="55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f"
Sep 17 16:27:22.015856 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.015799    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f"} err="failed to get container status \"55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f\": rpc error: code = NotFound desc = could not find container \"55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f\": container with ID starting with 55b1ccc7d9edeeea14079584e243b2f0ebb574043c61410abba7d7a0a0b1114f not found: ID does not exist"
Sep 17 16:27:22.015856 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.015813    2585 scope.go:117] "RemoveContainer" containerID="2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b"
Sep 17 16:27:22.016063 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:27:22.016042    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b\": container with ID starting with 2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b not found: ID does not exist" containerID="2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b"
Sep 17 16:27:22.016126 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.016068    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b"} err="failed to get container status \"2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b\": rpc error: code = NotFound desc = could not find container \"2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b\": container with ID starting with 2b91db4e6f6f3100358629bbd4a328f2cd1609d589cbd5b9d96581ed7c7f6d2b not found: ID does not exist"
Sep 17 16:27:22.016262 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.016245    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-4bnsp"]
Sep 17 16:27:22.829438 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:22.829401    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:27:23.782763 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:23.782722    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" path="/var/lib/kubelet/pods/5562595b-3732-4b02-bb1a-d71f0bc2a3a6/volumes"
Sep 17 16:27:32.829965 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:32.829915    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:27:33.112527 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:33.112435    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"]
Sep 17 16:27:33.112866 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:33.112708    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" containerID="cri-o://e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0" gracePeriod=30
Sep 17 16:27:33.112866 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:33.112803    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kube-rbac-proxy" containerID="cri-o://f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d" gracePeriod=30
Sep 17 16:27:34.056531 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:34.056497    2585 generic.go:356] "Generic (PLEG): container finished" podID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerID="f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d" exitCode=2
Sep 17 16:27:34.056910 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:34.056547    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" event={"ID":"b3cdb019-77bf-49c0-8e87-152aab55b231","Type":"ContainerDied","Data":"f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d"}
Sep 17 16:27:36.578035 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:36.577993    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 16:27:36.583924 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:36.583896    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:27:38.274675 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.274646    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:27:38.318308 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.318211    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3cdb019-77bf-49c0-8e87-152aab55b231-proxy-tls\") pod \"b3cdb019-77bf-49c0-8e87-152aab55b231\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") "
Sep 17 16:27:38.318308 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.318251    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-cabundle-cert\") pod \"b3cdb019-77bf-49c0-8e87-152aab55b231\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") "
Sep 17 16:27:38.318514 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.318310    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3cdb019-77bf-49c0-8e87-152aab55b231-kserve-provision-location\") pod \"b3cdb019-77bf-49c0-8e87-152aab55b231\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") "
Sep 17 16:27:38.318514 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.318348    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-rbac-proxy-sar-config\") pod \"b3cdb019-77bf-49c0-8e87-152aab55b231\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") "
Sep 17 16:27:38.318514 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.318375    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wssm\" (UniqueName: \"kubernetes.io/projected/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-api-access-8wssm\") pod \"b3cdb019-77bf-49c0-8e87-152aab55b231\" (UID: \"b3cdb019-77bf-49c0-8e87-152aab55b231\") "
Sep 17 16:27:38.319123 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.318661    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3cdb019-77bf-49c0-8e87-152aab55b231" (UID: "b3cdb019-77bf-49c0-8e87-152aab55b231"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:38.319123 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.318673    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3cdb019-77bf-49c0-8e87-152aab55b231" (UID: "b3cdb019-77bf-49c0-8e87-152aab55b231"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:38.319123 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.318928    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cdb019-77bf-49c0-8e87-152aab55b231-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3cdb019-77bf-49c0-8e87-152aab55b231" (UID: "b3cdb019-77bf-49c0-8e87-152aab55b231"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:38.320909 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.320887    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-api-access-8wssm" (OuterVolumeSpecName: "kube-api-access-8wssm") pod "b3cdb019-77bf-49c0-8e87-152aab55b231" (UID: "b3cdb019-77bf-49c0-8e87-152aab55b231"). InnerVolumeSpecName "kube-api-access-8wssm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:27:38.321058 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.321043    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cdb019-77bf-49c0-8e87-152aab55b231-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3cdb019-77bf-49c0-8e87-152aab55b231" (UID: "b3cdb019-77bf-49c0-8e87-152aab55b231"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:27:38.382684 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.382654    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"]
Sep 17 16:27:38.383160 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383146    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="storage-initializer"
Sep 17 16:27:38.383204 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383164    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="storage-initializer"
Sep 17 16:27:38.383204 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383179    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kube-rbac-proxy"
Sep 17 16:27:38.383204 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383188    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kube-rbac-proxy"
Sep 17 16:27:38.383309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383205    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kube-rbac-proxy"
Sep 17 16:27:38.383309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383214    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kube-rbac-proxy"
Sep 17 16:27:38.383309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383236    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container"
Sep 17 16:27:38.383309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383244    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container"
Sep 17 16:27:38.383309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383258    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container"
Sep 17 16:27:38.383309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383265    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container"
Sep 17 16:27:38.383309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383274    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="storage-initializer"
Sep 17 16:27:38.383309 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383298    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="storage-initializer"
Sep 17 16:27:38.383560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383371    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kube-rbac-proxy"
Sep 17 16:27:38.383560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383385    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kserve-container"
Sep 17 16:27:38.383560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383393    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5562595b-3732-4b02-bb1a-d71f0bc2a3a6" containerName="kube-rbac-proxy"
Sep 17 16:27:38.383560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.383405    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerName="kserve-container"
Sep 17 16:27:38.387090 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.387073    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.389653 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.389635    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 16:27:38.389775 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.389750    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 16:27:38.396591 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.396571    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"]
Sep 17 16:27:38.419888 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.419867    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbwg\" (UniqueName: \"kubernetes.io/projected/68bd910c-be14-4454-bbe7-56c23e609932-kube-api-access-sgbwg\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.419975 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.419915    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68bd910c-be14-4454-bbe7-56c23e609932-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.420029 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.419979    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.420065 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.420027    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.420140 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.420119    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.420217 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.420203    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3cdb019-77bf-49c0-8e87-152aab55b231-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.420256 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.420224    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.420256 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.420240    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3cdb019-77bf-49c0-8e87-152aab55b231-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.420331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.420254    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.420331 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.420269    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wssm\" (UniqueName: \"kubernetes.io/projected/b3cdb019-77bf-49c0-8e87-152aab55b231-kube-api-access-8wssm\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.521397 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.521367    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68bd910c-be14-4454-bbe7-56c23e609932-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.521539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.521417    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.521539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.521440    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.521539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.521494    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.521539 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.521534    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbwg\" (UniqueName: \"kubernetes.io/projected/68bd910c-be14-4454-bbe7-56c23e609932-kube-api-access-sgbwg\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.521762 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:27:38.521542    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 16:27:38.521762 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:27:38.521621    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls podName:68bd910c-be14-4454-bbe7-56c23e609932 nodeName:}" failed. No retries permitted until 2026-09-17 16:27:39.021598295 +0000 UTC m=+2025.287113289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls") pod "isvc-canary-promote-predictor-5fd5fb8776-9z5k8" (UID: "68bd910c-be14-4454-bbe7-56c23e609932") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 16:27:38.521887 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.521792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68bd910c-be14-4454-bbe7-56c23e609932-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.522164 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.522143    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.522243 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.522152    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:38.530563 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:38.530544    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbwg\" (UniqueName: \"kubernetes.io/projected/68bd910c-be14-4454-bbe7-56c23e609932-kube-api-access-sgbwg\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:39.025020 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.024975    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:39.027533 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.027514    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-9z5k8\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:39.080529 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.080499    2585 generic.go:356] "Generic (PLEG): container finished" podID="b3cdb019-77bf-49c0-8e87-152aab55b231" containerID="e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0" exitCode=0
Sep 17 16:27:39.080668 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.080566    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"
Sep 17 16:27:39.080668 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.080581    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" event={"ID":"b3cdb019-77bf-49c0-8e87-152aab55b231","Type":"ContainerDied","Data":"e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0"}
Sep 17 16:27:39.080668 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.080642    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz" event={"ID":"b3cdb019-77bf-49c0-8e87-152aab55b231","Type":"ContainerDied","Data":"c12e3362f67040348aa529496657264069bd8d637135f025bfdc0a9dc35e4d2e"}
Sep 17 16:27:39.080780 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.080675    2585 scope.go:117] "RemoveContainer" containerID="f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d"
Sep 17 16:27:39.089333 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.089316    2585 scope.go:117] "RemoveContainer" containerID="e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0"
Sep 17 16:27:39.097399 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.097377    2585 scope.go:117] "RemoveContainer" containerID="626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327"
Sep 17 16:27:39.104691 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.104673    2585 scope.go:117] "RemoveContainer" containerID="f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d"
Sep 17 16:27:39.104989 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:27:39.104968    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d\": container with ID starting with f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d not found: ID does not exist" containerID="f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d"
Sep 17 16:27:39.105040 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.104998    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d"} err="failed to get container status \"f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d\": rpc error: code = NotFound desc = could not find container \"f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d\": container with ID starting with f37d0813573ce0137db900af1681d003b23c3f862b41ec098b4d6cfbc00c489d not found: ID does not exist"
Sep 17 16:27:39.105040 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.105014    2585 scope.go:117] "RemoveContainer" containerID="e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0"
Sep 17 16:27:39.105270 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:27:39.105250    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0\": container with ID starting with e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0 not found: ID does not exist" containerID="e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0"
Sep 17 16:27:39.105335 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.105277    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0"} err="failed to get container status \"e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0\": rpc error: code = NotFound desc = could not find container \"e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0\": container with ID starting with e4e5197cc28ce92c5e1308ce7024fa8c1c8fc6677e9396151915e3911534d9e0 not found: ID does not exist"
Sep 17 16:27:39.105335 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.105319    2585 scope.go:117] "RemoveContainer" containerID="626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327"
Sep 17 16:27:39.105571 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:27:39.105556    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327\": container with ID starting with 626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327 not found: ID does not exist" containerID="626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327"
Sep 17 16:27:39.105621 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.105575    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327"} err="failed to get container status \"626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327\": rpc error: code = NotFound desc = could not find container \"626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327\": container with ID starting with 626608f96fce334b27fb485c57c39a8a606cd1734f0f04304bd34a6d8bfff327 not found: ID does not exist"
Sep 17 16:27:39.105621 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.105580    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"]
Sep 17 16:27:39.110758 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.110737    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-nglgz"]
Sep 17 16:27:39.299585 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.299511    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:39.574806 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.574780    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"]
Sep 17 16:27:39.577235 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:27:39.577207    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bd910c_be14_4454_bbe7_56c23e609932.slice/crio-62f7c90ca025d1ed96926db412b930412b504dfb1efc6d81fa0071321f122f5d WatchSource:0}: Error finding container 62f7c90ca025d1ed96926db412b930412b504dfb1efc6d81fa0071321f122f5d: Status 404 returned error can't find the container with id 62f7c90ca025d1ed96926db412b930412b504dfb1efc6d81fa0071321f122f5d
Sep 17 16:27:39.783095 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:39.783057    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3cdb019-77bf-49c0-8e87-152aab55b231" path="/var/lib/kubelet/pods/b3cdb019-77bf-49c0-8e87-152aab55b231/volumes"
Sep 17 16:27:40.089702 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:40.089663    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" event={"ID":"68bd910c-be14-4454-bbe7-56c23e609932","Type":"ContainerStarted","Data":"d0398bf35bf964e3e8bd2e46fb69159143374fd6373c573aff3fd899006659b2"}
Sep 17 16:27:40.089880 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:40.089740    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" event={"ID":"68bd910c-be14-4454-bbe7-56c23e609932","Type":"ContainerStarted","Data":"62f7c90ca025d1ed96926db412b930412b504dfb1efc6d81fa0071321f122f5d"}
Sep 17 16:27:42.829359 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:42.829319    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:27:44.109489 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:44.109454    2585 generic.go:356] "Generic (PLEG): container finished" podID="68bd910c-be14-4454-bbe7-56c23e609932" containerID="d0398bf35bf964e3e8bd2e46fb69159143374fd6373c573aff3fd899006659b2" exitCode=0
Sep 17 16:27:44.109925 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:44.109520    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" event={"ID":"68bd910c-be14-4454-bbe7-56c23e609932","Type":"ContainerDied","Data":"d0398bf35bf964e3e8bd2e46fb69159143374fd6373c573aff3fd899006659b2"}
Sep 17 16:27:45.119883 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:45.119850    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" event={"ID":"68bd910c-be14-4454-bbe7-56c23e609932","Type":"ContainerStarted","Data":"11f9dbd9e6f215b8ca277d3ede7503af395f9bcf9b9b3e36ac8b50f710456905"}
Sep 17 16:27:45.120271 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:45.119912    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" event={"ID":"68bd910c-be14-4454-bbe7-56c23e609932","Type":"ContainerStarted","Data":"1d4d613e251d14516b63152517bb551e08b67d1349dc11541762e5534fadd026"}
Sep 17 16:27:45.120271 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:45.120101    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:45.140946 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:45.140894    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podStartSLOduration=3.310047337 podStartE2EDuration="7.140876761s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.830829424s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:27:45.139780124 +0000 UTC m=+2031.405295124" watchObservedRunningTime="2026-09-17 16:27:45.140876761 +0000 UTC m=+2031.406391764"
Sep 17 16:27:46.124550 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:46.124517    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:46.125512 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:46.125485    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:27:47.128971 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:47.128931    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:27:52.133116 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:52.133085    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:27:52.133636 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:52.133560    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:27:52.829690 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:27:52.829648    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:28:02.134023 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:02.133973    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:28:02.830433 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:02.830396    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:28:09.874930 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:09.874834    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"]
Sep 17 16:28:09.875427 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:09.875142    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" containerID="cri-o://5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19" gracePeriod=30
Sep 17 16:28:09.875427 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:09.875174    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kube-rbac-proxy" containerID="cri-o://38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094" gracePeriod=30
Sep 17 16:28:10.234592 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:10.234489    2585 generic.go:356] "Generic (PLEG): container finished" podID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerID="38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094" exitCode=2
Sep 17 16:28:10.234592 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:10.234561    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" event={"ID":"2e2e5405-058a-4a65-ba74-2356783a3e76","Type":"ContainerDied","Data":"38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094"}
Sep 17 16:28:12.134295 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:12.134240    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:28:12.823259 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:12.823218    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 16:28:12.829768 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:12.829743    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:28:15.029895 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.029871    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:28:15.043724 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.043699    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e2e5405-058a-4a65-ba74-2356783a3e76-proxy-tls\") pod \"2e2e5405-058a-4a65-ba74-2356783a3e76\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") "
Sep 17 16:28:15.043898 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.043759    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgxdt\" (UniqueName: \"kubernetes.io/projected/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-api-access-sgxdt\") pod \"2e2e5405-058a-4a65-ba74-2356783a3e76\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") "
Sep 17 16:28:15.043898 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.043808    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-rbac-proxy-sar-config\") pod \"2e2e5405-058a-4a65-ba74-2356783a3e76\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") "
Sep 17 16:28:15.043898 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.043855    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-cabundle-cert\") pod \"2e2e5405-058a-4a65-ba74-2356783a3e76\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") "
Sep 17 16:28:15.044070 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.043919    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2e5405-058a-4a65-ba74-2356783a3e76-kserve-provision-location\") pod \"2e2e5405-058a-4a65-ba74-2356783a3e76\" (UID: \"2e2e5405-058a-4a65-ba74-2356783a3e76\") "
Sep 17 16:28:15.044239 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.044207    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e2e5405-058a-4a65-ba74-2356783a3e76" (UID: "2e2e5405-058a-4a65-ba74-2356783a3e76"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:15.044382 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.044264    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e2e5405-058a-4a65-ba74-2356783a3e76" (UID: "2e2e5405-058a-4a65-ba74-2356783a3e76"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:15.044626 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.044468    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2e5405-058a-4a65-ba74-2356783a3e76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e2e5405-058a-4a65-ba74-2356783a3e76" (UID: "2e2e5405-058a-4a65-ba74-2356783a3e76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:15.046054 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.046025    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2e5405-058a-4a65-ba74-2356783a3e76-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e2e5405-058a-4a65-ba74-2356783a3e76" (UID: "2e2e5405-058a-4a65-ba74-2356783a3e76"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:15.046154 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.046097    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-api-access-sgxdt" (OuterVolumeSpecName: "kube-api-access-sgxdt") pod "2e2e5405-058a-4a65-ba74-2356783a3e76" (UID: "2e2e5405-058a-4a65-ba74-2356783a3e76"). InnerVolumeSpecName "kube-api-access-sgxdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:15.145450 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.145366    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e2e5405-058a-4a65-ba74-2356783a3e76-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.145450 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.145397    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgxdt\" (UniqueName: \"kubernetes.io/projected/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-api-access-sgxdt\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.145450 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.145407    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.145450 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.145416    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2e5405-058a-4a65-ba74-2356783a3e76-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.145450 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.145425    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2e5405-058a-4a65-ba74-2356783a3e76-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.260732 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.260698    2585 generic.go:356] "Generic (PLEG): container finished" podID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerID="5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19" exitCode=0
Sep 17 16:28:15.260916 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.260765    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" event={"ID":"2e2e5405-058a-4a65-ba74-2356783a3e76","Type":"ContainerDied","Data":"5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19"}
Sep 17 16:28:15.260916 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.260839    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79" event={"ID":"2e2e5405-058a-4a65-ba74-2356783a3e76","Type":"ContainerDied","Data":"8961752eefaa4d6cf16b71e288de7d2575ce3959a07c288f8db88752d92e496e"}
Sep 17 16:28:15.260916 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.260765    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"
Sep 17 16:28:15.260916 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.260864    2585 scope.go:117] "RemoveContainer" containerID="38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094"
Sep 17 16:28:15.272696 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.272396    2585 scope.go:117] "RemoveContainer" containerID="5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19"
Sep 17 16:28:15.280956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.280931    2585 scope.go:117] "RemoveContainer" containerID="a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360"
Sep 17 16:28:15.286984 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.286958    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"]
Sep 17 16:28:15.290109 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.290047    2585 scope.go:117] "RemoveContainer" containerID="38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094"
Sep 17 16:28:15.290631 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:28:15.290603    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094\": container with ID starting with 38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094 not found: ID does not exist" containerID="38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094"
Sep 17 16:28:15.290733 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.290642    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094"} err="failed to get container status \"38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094\": rpc error: code = NotFound desc = could not find container \"38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094\": container with ID starting with 38ab03b3ebbf0a73a336a71e3f815f0d7ebb2eaac5a54fbbecf9b6373b9bf094 not found: ID does not exist"
Sep 17 16:28:15.290733 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.290669    2585 scope.go:117] "RemoveContainer" containerID="5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19"
Sep 17 16:28:15.291045 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:28:15.291025    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19\": container with ID starting with 5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19 not found: ID does not exist" containerID="5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19"
Sep 17 16:28:15.291105 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.291053    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19"} err="failed to get container status \"5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19\": rpc error: code = NotFound desc = could not find container \"5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19\": container with ID starting with 5069710a70fdfcc6c68d6bc731af2e01eef757fac0c660b5e85b9abfa5065e19 not found: ID does not exist"
Sep 17 16:28:15.291105 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.291069    2585 scope.go:117] "RemoveContainer" containerID="a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360"
Sep 17 16:28:15.291396 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:28:15.291371    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360\": container with ID starting with a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360 not found: ID does not exist" containerID="a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360"
Sep 17 16:28:15.291523 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.291401    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360"} err="failed to get container status \"a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360\": rpc error: code = NotFound desc = could not find container \"a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360\": container with ID starting with a4899994f7fee0374e65bdbf5ad7f362da5062cff8e6c4f493cae82101700360 not found: ID does not exist"
Sep 17 16:28:15.292304 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.292261    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-75fcdd79b8-8pm79"]
Sep 17 16:28:15.782022 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:15.781985    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" path="/var/lib/kubelet/pods/2e2e5405-058a-4a65-ba74-2356783a3e76/volumes"
Sep 17 16:28:22.133562 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:22.133520    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:28:32.133715 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:32.133665    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:28:42.133548 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:42.133501    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:28:52.134176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:52.134136    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:28:55.779501 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:28:55.779455    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:29:02.229708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.229671    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"]
Sep 17 16:29:02.230084 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.230017    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="storage-initializer"
Sep 17 16:29:02.230084 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.230030    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="storage-initializer"
Sep 17 16:29:02.230084 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.230039    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container"
Sep 17 16:29:02.230084 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.230045    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container"
Sep 17 16:29:02.230084 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.230052    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kube-rbac-proxy"
Sep 17 16:29:02.230084 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.230058    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kube-rbac-proxy"
Sep 17 16:29:02.230278 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.230110    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kube-rbac-proxy"
Sep 17 16:29:02.230278 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.230117    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2e5405-058a-4a65-ba74-2356783a3e76" containerName="kserve-container"
Sep 17 16:29:02.233895 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.233879    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.236437 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.236411    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:29:02.236562 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.236440    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 16:29:02.245070 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.245040    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"]
Sep 17 16:29:02.342410 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.342368    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rncfq\" (UniqueName: \"kubernetes.io/projected/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-api-access-rncfq\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.342410 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.342414    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.342687 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.342432    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.342687 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.342505    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.342687 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.342578    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.443517 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.443480    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rncfq\" (UniqueName: \"kubernetes.io/projected/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-api-access-rncfq\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.443517 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.443521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.443763 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.443539    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.443763 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.443564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.443763 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.443633    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.444062 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.444042    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.444167 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.444149    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.444264 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.444244    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.446244 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.446226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.453012 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.452988    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rncfq\" (UniqueName: \"kubernetes.io/projected/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-api-access-rncfq\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-x59pg\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.544527 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.544485    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:02.824544 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:02.824510    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"]
Sep 17 16:29:02.826959 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:29:02.826934    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6dcfc8_7c96_4fa3_80e6_5055fd90d35b.slice/crio-752c3d8e3ac1edbf09b00b56d0a3eddf0fecffdb2050d7afc25eb152bcb3163b WatchSource:0}: Error finding container 752c3d8e3ac1edbf09b00b56d0a3eddf0fecffdb2050d7afc25eb152bcb3163b: Status 404 returned error can't find the container with id 752c3d8e3ac1edbf09b00b56d0a3eddf0fecffdb2050d7afc25eb152bcb3163b
Sep 17 16:29:03.474683 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:03.474641    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" event={"ID":"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b","Type":"ContainerStarted","Data":"f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df"}
Sep 17 16:29:03.475154 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:03.474699    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" event={"ID":"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b","Type":"ContainerStarted","Data":"752c3d8e3ac1edbf09b00b56d0a3eddf0fecffdb2050d7afc25eb152bcb3163b"}
Sep 17 16:29:04.644113 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:04.644079    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"]
Sep 17 16:29:04.644644 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:04.644400    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container" containerID="cri-o://1d4d613e251d14516b63152517bb551e08b67d1349dc11541762e5534fadd026" gracePeriod=30
Sep 17 16:29:04.644644 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:04.644447    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kube-rbac-proxy" containerID="cri-o://11f9dbd9e6f215b8ca277d3ede7503af395f9bcf9b9b3e36ac8b50f710456905" gracePeriod=30
Sep 17 16:29:04.645307 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:04.645268    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:29:05.484956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:05.484924    2585 generic.go:356] "Generic (PLEG): container finished" podID="68bd910c-be14-4454-bbe7-56c23e609932" containerID="11f9dbd9e6f215b8ca277d3ede7503af395f9bcf9b9b3e36ac8b50f710456905" exitCode=2
Sep 17 16:29:05.485146 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:05.484998    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" event={"ID":"68bd910c-be14-4454-bbe7-56c23e609932","Type":"ContainerDied","Data":"11f9dbd9e6f215b8ca277d3ede7503af395f9bcf9b9b3e36ac8b50f710456905"}
Sep 17 16:29:07.129376 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:07.129318    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 16:29:07.497071 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:07.496981    2585 generic.go:356] "Generic (PLEG): container finished" podID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerID="f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df" exitCode=0
Sep 17 16:29:07.497071 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:07.497052    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" event={"ID":"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b","Type":"ContainerDied","Data":"f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df"}
Sep 17 16:29:07.498274 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:07.498260    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:29:08.407772 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.407734    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"]
Sep 17 16:29:08.413580 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.413551    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.416655 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.416625    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 16:29:08.428957 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.428932    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"]
Sep 17 16:29:08.493136 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.493084    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.493136 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.493132    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.493403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.493189    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.493403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.493305    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbn7\" (UniqueName: \"kubernetes.io/projected/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-api-access-6jbn7\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.493403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.493348    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d15e912-a4b0-42a6-a961-e6db715c4ba4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.594508 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.594465    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d15e912-a4b0-42a6-a961-e6db715c4ba4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.594696 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.594575    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.594762 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.594697    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.594762 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.594729    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.594881 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.594807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbn7\" (UniqueName: \"kubernetes.io/projected/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-api-access-6jbn7\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.594995 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.594950    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.595534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.595472    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.595534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.595482    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.597612 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.597592    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d15e912-a4b0-42a6-a961-e6db715c4ba4-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.605755 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.605730    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbn7\" (UniqueName: \"kubernetes.io/projected/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-api-access-6jbn7\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:08.728071 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:08.727983    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:09.355783 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.355561    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"]
Sep 17 16:29:09.358897 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:29:09.358862    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d15e912_a4b0_42a6_a961_e6db715c4ba4.slice/crio-529577d19aa537f3a1a0c0212d66ed9c3a8a822dddf1f410572fc1e46f6a8041 WatchSource:0}: Error finding container 529577d19aa537f3a1a0c0212d66ed9c3a8a822dddf1f410572fc1e46f6a8041: Status 404 returned error can't find the container with id 529577d19aa537f3a1a0c0212d66ed9c3a8a822dddf1f410572fc1e46f6a8041
Sep 17 16:29:09.511363 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.510627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" event={"ID":"2d15e912-a4b0-42a6-a961-e6db715c4ba4","Type":"ContainerStarted","Data":"6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195"}
Sep 17 16:29:09.511363 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.510701    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" event={"ID":"2d15e912-a4b0-42a6-a961-e6db715c4ba4","Type":"ContainerStarted","Data":"529577d19aa537f3a1a0c0212d66ed9c3a8a822dddf1f410572fc1e46f6a8041"}
Sep 17 16:29:09.514243 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.514212    2585 generic.go:356] "Generic (PLEG): container finished" podID="68bd910c-be14-4454-bbe7-56c23e609932" containerID="1d4d613e251d14516b63152517bb551e08b67d1349dc11541762e5534fadd026" exitCode=0
Sep 17 16:29:09.514243 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.514247    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" event={"ID":"68bd910c-be14-4454-bbe7-56c23e609932","Type":"ContainerDied","Data":"1d4d613e251d14516b63152517bb551e08b67d1349dc11541762e5534fadd026"}
Sep 17 16:29:09.631828 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.631802    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:29:09.705920 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.705890    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-cabundle-cert\") pod \"68bd910c-be14-4454-bbe7-56c23e609932\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") "
Sep 17 16:29:09.706105 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.705957    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgbwg\" (UniqueName: \"kubernetes.io/projected/68bd910c-be14-4454-bbe7-56c23e609932-kube-api-access-sgbwg\") pod \"68bd910c-be14-4454-bbe7-56c23e609932\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") "
Sep 17 16:29:09.706105 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.705989    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68bd910c-be14-4454-bbe7-56c23e609932-kserve-provision-location\") pod \"68bd910c-be14-4454-bbe7-56c23e609932\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") "
Sep 17 16:29:09.706105 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.706014    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls\") pod \"68bd910c-be14-4454-bbe7-56c23e609932\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") "
Sep 17 16:29:09.706105 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.706065    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-kube-rbac-proxy-sar-config\") pod \"68bd910c-be14-4454-bbe7-56c23e609932\" (UID: \"68bd910c-be14-4454-bbe7-56c23e609932\") "
Sep 17 16:29:09.706349 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.706211    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68bd910c-be14-4454-bbe7-56c23e609932" (UID: "68bd910c-be14-4454-bbe7-56c23e609932"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:09.706410 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.706394    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:29:09.706484 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.706453    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bd910c-be14-4454-bbe7-56c23e609932-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68bd910c-be14-4454-bbe7-56c23e609932" (UID: "68bd910c-be14-4454-bbe7-56c23e609932"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:29:09.706697 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.706657    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68bd910c-be14-4454-bbe7-56c23e609932" (UID: "68bd910c-be14-4454-bbe7-56c23e609932"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:09.708585 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.708559    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bd910c-be14-4454-bbe7-56c23e609932-kube-api-access-sgbwg" (OuterVolumeSpecName: "kube-api-access-sgbwg") pod "68bd910c-be14-4454-bbe7-56c23e609932" (UID: "68bd910c-be14-4454-bbe7-56c23e609932"). InnerVolumeSpecName "kube-api-access-sgbwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:29:09.708585 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.708571    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68bd910c-be14-4454-bbe7-56c23e609932" (UID: "68bd910c-be14-4454-bbe7-56c23e609932"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:29:09.807406 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.807375    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbwg\" (UniqueName: \"kubernetes.io/projected/68bd910c-be14-4454-bbe7-56c23e609932-kube-api-access-sgbwg\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:29:09.807551 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.807409    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68bd910c-be14-4454-bbe7-56c23e609932-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:29:09.807551 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.807425    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68bd910c-be14-4454-bbe7-56c23e609932-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:29:09.807551 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:09.807440    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68bd910c-be14-4454-bbe7-56c23e609932-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:29:10.523507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:10.523475    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"
Sep 17 16:29:10.523507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:10.523490    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8" event={"ID":"68bd910c-be14-4454-bbe7-56c23e609932","Type":"ContainerDied","Data":"62f7c90ca025d1ed96926db412b930412b504dfb1efc6d81fa0071321f122f5d"}
Sep 17 16:29:10.524010 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:10.523568    2585 scope.go:117] "RemoveContainer" containerID="11f9dbd9e6f215b8ca277d3ede7503af395f9bcf9b9b3e36ac8b50f710456905"
Sep 17 16:29:10.545243 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:10.545216    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"]
Sep 17 16:29:10.552985 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:10.552965    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-9z5k8"]
Sep 17 16:29:11.148975 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.148953    2585 scope.go:117] "RemoveContainer" containerID="1d4d613e251d14516b63152517bb551e08b67d1349dc11541762e5534fadd026"
Sep 17 16:29:11.197053 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.197034    2585 scope.go:117] "RemoveContainer" containerID="d0398bf35bf964e3e8bd2e46fb69159143374fd6373c573aff3fd899006659b2"
Sep 17 16:29:11.531097 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.531069    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" event={"ID":"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b","Type":"ContainerStarted","Data":"f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d"}
Sep 17 16:29:11.531474 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.531126    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" event={"ID":"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b","Type":"ContainerStarted","Data":"d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252"}
Sep 17 16:29:11.531474 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.531428    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:11.531586 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.531482    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:11.532983 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.532956    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:29:11.553658 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.553616    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podStartSLOduration=1.5270383 podStartE2EDuration="9.55360256s" totalImagesPullingTime="3.748981418s" totalInitContainerRuntime="4.277582842s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:29:11.552161907 +0000 UTC m=+2117.817676908" watchObservedRunningTime="2026-09-17 16:29:11.55360256 +0000 UTC m=+2117.819117559"
Sep 17 16:29:11.783368 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:11.783275    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68bd910c-be14-4454-bbe7-56c23e609932" path="/var/lib/kubelet/pods/68bd910c-be14-4454-bbe7-56c23e609932/volumes"
Sep 17 16:29:12.535876 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:12.535837    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:29:13.542852 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:13.542823    2585 generic.go:356] "Generic (PLEG): container finished" podID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerID="6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195" exitCode=0
Sep 17 16:29:13.543190 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:13.542892    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" event={"ID":"2d15e912-a4b0-42a6-a961-e6db715c4ba4","Type":"ContainerDied","Data":"6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195"}
Sep 17 16:29:14.553045 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:14.553007    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" event={"ID":"2d15e912-a4b0-42a6-a961-e6db715c4ba4","Type":"ContainerStarted","Data":"d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2"}
Sep 17 16:29:14.553507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:14.553067    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" event={"ID":"2d15e912-a4b0-42a6-a961-e6db715c4ba4","Type":"ContainerStarted","Data":"c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad"}
Sep 17 16:29:14.553507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:14.553267    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:14.578926 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:14.578867    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podStartSLOduration=2.513374232 podStartE2EDuration="6.578850057s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.065475825s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:29:14.576102982 +0000 UTC m=+2120.841617982" watchObservedRunningTime="2026-09-17 16:29:14.578850057 +0000 UTC m=+2120.844365129"
Sep 17 16:29:15.558224 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:15.558193    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:15.559589 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:15.559558    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:29:16.563156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:16.563112    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:29:17.540442 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:17.540410    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:17.541100 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:17.541072    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:29:21.568074 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:21.568039    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:29:21.568681 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:21.568508    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:29:25.902003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:25.901974    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:29:25.903348 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:25.903327    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:29:27.541997 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:27.541963    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:29:31.568495 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:31.568444    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:29:41.569536 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:41.569451    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:29:44.062984 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:44.062950    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"]
Sep 17 16:29:44.065472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:44.063193    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kserve-container" containerID="cri-o://d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252" gracePeriod=30
Sep 17 16:29:44.065472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:44.063222    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy" containerID="cri-o://f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d" gracePeriod=30
Sep 17 16:29:44.697588 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:44.697555    2585 generic.go:356] "Generic (PLEG): container finished" podID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerID="f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d" exitCode=2
Sep 17 16:29:44.697772 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:44.697627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" event={"ID":"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b","Type":"ContainerDied","Data":"f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d"}
Sep 17 16:29:47.536116 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:47.536081    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 16:29:51.568911 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:51.568871    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:29:52.536731 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:52.536694    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 16:29:57.536813 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:57.536771    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 16:29:57.537268 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:29:57.536916    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:30:01.569022 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:01.568983    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:30:02.536959 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:02.536915    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 16:30:07.536472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:07.536428    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 16:30:11.568661 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:11.568623    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:30:12.536034 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:12.535987    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 16:30:14.706905 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.706880    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:30:14.758995 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.758970    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-proxy-tls\") pod \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") "
Sep 17 16:30:14.759116 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.759002    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-cabundle-cert\") pod \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") "
Sep 17 16:30:14.759116 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.759036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kserve-provision-location\") pod \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") "
Sep 17 16:30:14.759200 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.759134    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rncfq\" (UniqueName: \"kubernetes.io/projected/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-api-access-rncfq\") pod \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") "
Sep 17 16:30:14.759253 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.759201    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-rbac-proxy-sar-config\") pod \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\" (UID: \"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b\") "
Sep 17 16:30:14.759427 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.759394    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" (UID: "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:30:14.759560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.759522    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:14.759625 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.759567    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" (UID: "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:30:14.761336 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.761309    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" (UID: "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:30:14.761431 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.761327    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-api-access-rncfq" (OuterVolumeSpecName: "kube-api-access-rncfq") pod "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" (UID: "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b"). InnerVolumeSpecName "kube-api-access-rncfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:30:14.769139 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.769111    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" (UID: "0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:30:14.832890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.832812    2585 generic.go:356] "Generic (PLEG): container finished" podID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerID="d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252" exitCode=137
Sep 17 16:30:14.832890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.832864    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"
Sep 17 16:30:14.833035 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.832865    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" event={"ID":"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b","Type":"ContainerDied","Data":"d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252"}
Sep 17 16:30:14.833035 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.833002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg" event={"ID":"0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b","Type":"ContainerDied","Data":"752c3d8e3ac1edbf09b00b56d0a3eddf0fecffdb2050d7afc25eb152bcb3163b"}
Sep 17 16:30:14.833035 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.833035    2585 scope.go:117] "RemoveContainer" containerID="f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d"
Sep 17 16:30:14.843301 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.843271    2585 scope.go:117] "RemoveContainer" containerID="d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252"
Sep 17 16:30:14.851035 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.851017    2585 scope.go:117] "RemoveContainer" containerID="f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df"
Sep 17 16:30:14.854777 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.854755    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"]
Sep 17 16:30:14.858520 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.858499    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-x59pg"]
Sep 17 16:30:14.859260 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859247    2585 scope.go:117] "RemoveContainer" containerID="f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d"
Sep 17 16:30:14.859577 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:30:14.859556    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d\": container with ID starting with f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d not found: ID does not exist" containerID="f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d"
Sep 17 16:30:14.859657 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859585    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d"} err="failed to get container status \"f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d\": rpc error: code = NotFound desc = could not find container \"f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d\": container with ID starting with f059a8e68487917414bf092414515d886e4ad1b68bad65a2053b094252be6a7d not found: ID does not exist"
Sep 17 16:30:14.859657 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859605    2585 scope.go:117] "RemoveContainer" containerID="d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252"
Sep 17 16:30:14.859864 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:30:14.859842    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252\": container with ID starting with d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252 not found: ID does not exist" containerID="d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252"
Sep 17 16:30:14.859917 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859873    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252"} err="failed to get container status \"d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252\": rpc error: code = NotFound desc = could not find container \"d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252\": container with ID starting with d4a622ee743e2b8bbcdb5934d51664cf490b55e14c48d17dafd4426219247252 not found: ID does not exist"
Sep 17 16:30:14.859917 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859890    2585 scope.go:117] "RemoveContainer" containerID="f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df"
Sep 17 16:30:14.860015 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859918    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:14.860015 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859939    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:14.860015 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859956    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rncfq\" (UniqueName: \"kubernetes.io/projected/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-api-access-rncfq\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:14.860015 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.859970    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:14.860205 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:30:14.860182    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df\": container with ID starting with f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df not found: ID does not exist" containerID="f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df"
Sep 17 16:30:14.860247 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:14.860200    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df"} err="failed to get container status \"f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df\": rpc error: code = NotFound desc = could not find container \"f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df\": container with ID starting with f58e148645e0827811b4ee3660588261e7172787c64a7648cb59f17fd24646df not found: ID does not exist"
Sep 17 16:30:15.782713 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:15.782678    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" path="/var/lib/kubelet/pods/0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b/volumes"
Sep 17 16:30:21.568520 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:21.568480    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:30:31.569520 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:31.569478    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:30:38.514696 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:38.514661    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"]
Sep 17 16:30:38.515088 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:38.514927    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" containerID="cri-o://c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad" gracePeriod=30
Sep 17 16:30:38.515088 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:38.514972    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kube-rbac-proxy" containerID="cri-o://d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2" gracePeriod=30
Sep 17 16:30:38.945640 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:38.945540    2585 generic.go:356] "Generic (PLEG): container finished" podID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerID="d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2" exitCode=2
Sep 17 16:30:38.945640 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:38.945600    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" event={"ID":"2d15e912-a4b0-42a6-a961-e6db715c4ba4","Type":"ContainerDied","Data":"d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2"}
Sep 17 16:30:41.564401 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:41.564355    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 16:30:41.568777 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:41.568746    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:30:43.564142 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.564110    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:30:43.712755 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.712650    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-rbac-proxy-sar-config\") pod \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") "
Sep 17 16:30:43.712755 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.712710    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kserve-provision-location\") pod \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") "
Sep 17 16:30:43.712755 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.712751    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d15e912-a4b0-42a6-a961-e6db715c4ba4-proxy-tls\") pod \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") "
Sep 17 16:30:43.713026 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.712807    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-cabundle-cert\") pod \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") "
Sep 17 16:30:43.713026 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.712838    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jbn7\" (UniqueName: \"kubernetes.io/projected/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-api-access-6jbn7\") pod \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\" (UID: \"2d15e912-a4b0-42a6-a961-e6db715c4ba4\") "
Sep 17 16:30:43.713157 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.713114    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d15e912-a4b0-42a6-a961-e6db715c4ba4" (UID: "2d15e912-a4b0-42a6-a961-e6db715c4ba4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:30:43.713157 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.713147    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d15e912-a4b0-42a6-a961-e6db715c4ba4" (UID: "2d15e912-a4b0-42a6-a961-e6db715c4ba4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:30:43.713266 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.713196    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d15e912-a4b0-42a6-a961-e6db715c4ba4" (UID: "2d15e912-a4b0-42a6-a961-e6db715c4ba4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:30:43.715111 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.715088    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d15e912-a4b0-42a6-a961-e6db715c4ba4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d15e912-a4b0-42a6-a961-e6db715c4ba4" (UID: "2d15e912-a4b0-42a6-a961-e6db715c4ba4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:30:43.715199 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.715160    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-api-access-6jbn7" (OuterVolumeSpecName: "kube-api-access-6jbn7") pod "2d15e912-a4b0-42a6-a961-e6db715c4ba4" (UID: "2d15e912-a4b0-42a6-a961-e6db715c4ba4"). InnerVolumeSpecName "kube-api-access-6jbn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:30:43.813659 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.813623    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:43.813659 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.813652    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:43.813659 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.813663    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d15e912-a4b0-42a6-a961-e6db715c4ba4-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:43.813910 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.813673    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d15e912-a4b0-42a6-a961-e6db715c4ba4-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:43.813910 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.813681    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jbn7\" (UniqueName: \"kubernetes.io/projected/2d15e912-a4b0-42a6-a961-e6db715c4ba4-kube-api-access-6jbn7\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:30:43.849306 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849250    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"]
Sep 17 16:30:43.849602 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849590    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849603    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849616    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="storage-initializer"
Sep 17 16:30:43.849647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849622    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="storage-initializer"
Sep 17 16:30:43.849647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849631    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container"
Sep 17 16:30:43.849647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849637    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container"
Sep 17 16:30:43.849647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849645    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="storage-initializer"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849651    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="storage-initializer"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849657    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="storage-initializer"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849663    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="storage-initializer"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849670    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849675    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849682    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849688    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849694    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849698    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849704    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kserve-container"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849709    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kserve-container"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849757    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kserve-container"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849765    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849771    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849776    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6dcfc8-7c96-4fa3-80e6-5055fd90d35b" containerName="kserve-container"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849782    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerName="kube-rbac-proxy"
Sep 17 16:30:43.849831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.849789    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bd910c-be14-4454-bbe7-56c23e609932" containerName="kserve-container"
Sep 17 16:30:43.853234 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.853211    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:43.857596 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.857575    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 16:30:43.857826 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.857810    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 16:30:43.871456 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.871417    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"]
Sep 17 16:30:43.968756 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.968644    2585 generic.go:356] "Generic (PLEG): container finished" podID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" containerID="c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad" exitCode=0
Sep 17 16:30:43.968756 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.968723    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"
Sep 17 16:30:43.968756 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.968738    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" event={"ID":"2d15e912-a4b0-42a6-a961-e6db715c4ba4","Type":"ContainerDied","Data":"c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad"}
Sep 17 16:30:43.969046 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.968855    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs" event={"ID":"2d15e912-a4b0-42a6-a961-e6db715c4ba4","Type":"ContainerDied","Data":"529577d19aa537f3a1a0c0212d66ed9c3a8a822dddf1f410572fc1e46f6a8041"}
Sep 17 16:30:43.969046 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.968897    2585 scope.go:117] "RemoveContainer" containerID="d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2"
Sep 17 16:30:43.977891 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.977869    2585 scope.go:117] "RemoveContainer" containerID="c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad"
Sep 17 16:30:43.986357 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.986329    2585 scope.go:117] "RemoveContainer" containerID="6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195"
Sep 17 16:30:43.995400 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.995374    2585 scope.go:117] "RemoveContainer" containerID="d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2"
Sep 17 16:30:43.995775 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:30:43.995755    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2\": container with ID starting with d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2 not found: ID does not exist" containerID="d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2"
Sep 17 16:30:43.995845 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.995785    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2"} err="failed to get container status \"d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2\": rpc error: code = NotFound desc = could not find container \"d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2\": container with ID starting with d2f1dd1de7556e91b08a2265110f60a7651df8b91d7187f7adc96bacc641fca2 not found: ID does not exist"
Sep 17 16:30:43.995845 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.995804    2585 scope.go:117] "RemoveContainer" containerID="c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad"
Sep 17 16:30:43.996132 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:30:43.996113    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad\": container with ID starting with c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad not found: ID does not exist" containerID="c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad"
Sep 17 16:30:43.996174 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.996140    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad"} err="failed to get container status \"c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad\": rpc error: code = NotFound desc = could not find container \"c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad\": container with ID starting with c714b98488a84f83e6501b5d88de4ab7837b8e6db995f4b3ecdbb69d10d410ad not found: ID does not exist"
Sep 17 16:30:43.996174 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.996157    2585 scope.go:117] "RemoveContainer" containerID="6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195"
Sep 17 16:30:43.996517 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:30:43.996500    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195\": container with ID starting with 6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195 not found: ID does not exist" containerID="6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195"
Sep 17 16:30:43.996592 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:43.996520    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195"} err="failed to get container status \"6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195\": rpc error: code = NotFound desc = could not find container \"6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195\": container with ID starting with 6c2a32373ef48f29b2eed4cd7710090c0451ec8cd00cb147ed784ddb26394195 not found: ID does not exist"
Sep 17 16:30:44.002535 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.002503    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"]
Sep 17 16:30:44.010612 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.010579    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-hp4fs"]
Sep 17 16:30:44.015967 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.015937    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4174708-9d4c-4150-aa10-b07235c55a38-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.016107 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.015978    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4174708-9d4c-4150-aa10-b07235c55a38-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.016169 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.016100    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.016227 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.016172    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.016326 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.016235    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnn8r\" (UniqueName: \"kubernetes.io/projected/d4174708-9d4c-4150-aa10-b07235c55a38-kube-api-access-qnn8r\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.117009 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.116970    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnn8r\" (UniqueName: \"kubernetes.io/projected/d4174708-9d4c-4150-aa10-b07235c55a38-kube-api-access-qnn8r\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.117201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.117015    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4174708-9d4c-4150-aa10-b07235c55a38-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.117201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.117051    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4174708-9d4c-4150-aa10-b07235c55a38-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.117201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.117115    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.117201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.117165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.117514 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.117481    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4174708-9d4c-4150-aa10-b07235c55a38-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.117840 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.117823    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.117952 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.117931    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.119816 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.119792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4174708-9d4c-4150-aa10-b07235c55a38-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.128015 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.127988    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnn8r\" (UniqueName: \"kubernetes.io/projected/d4174708-9d4c-4150-aa10-b07235c55a38-kube-api-access-qnn8r\") pod \"isvc-canary-rollback-predictor-6b9dfd44-pmcdc\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.163554 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.163511    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:44.446823 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.446801    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"]
Sep 17 16:30:44.449428 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:30:44.449392    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4174708_9d4c_4150_aa10_b07235c55a38.slice/crio-9f3b71630e2148d5e4357ae87aba559d7ed4a2d5909eebd16cca3ec4b5831ac4 WatchSource:0}: Error finding container 9f3b71630e2148d5e4357ae87aba559d7ed4a2d5909eebd16cca3ec4b5831ac4: Status 404 returned error can't find the container with id 9f3b71630e2148d5e4357ae87aba559d7ed4a2d5909eebd16cca3ec4b5831ac4
Sep 17 16:30:44.976081 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.976044    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" event={"ID":"d4174708-9d4c-4150-aa10-b07235c55a38","Type":"ContainerStarted","Data":"50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536"}
Sep 17 16:30:44.976562 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:44.976103    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" event={"ID":"d4174708-9d4c-4150-aa10-b07235c55a38","Type":"ContainerStarted","Data":"9f3b71630e2148d5e4357ae87aba559d7ed4a2d5909eebd16cca3ec4b5831ac4"}
Sep 17 16:30:45.782532 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:45.782501    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d15e912-a4b0-42a6-a961-e6db715c4ba4" path="/var/lib/kubelet/pods/2d15e912-a4b0-42a6-a961-e6db715c4ba4/volumes"
Sep 17 16:30:48.996724 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:48.996690    2585 generic.go:356] "Generic (PLEG): container finished" podID="d4174708-9d4c-4150-aa10-b07235c55a38" containerID="50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536" exitCode=0
Sep 17 16:30:48.997090 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:48.996769    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" event={"ID":"d4174708-9d4c-4150-aa10-b07235c55a38","Type":"ContainerDied","Data":"50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536"}
Sep 17 16:30:50.009555 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:50.009514    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" event={"ID":"d4174708-9d4c-4150-aa10-b07235c55a38","Type":"ContainerStarted","Data":"8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78"}
Sep 17 16:30:50.009956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:50.009613    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" event={"ID":"d4174708-9d4c-4150-aa10-b07235c55a38","Type":"ContainerStarted","Data":"e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1"}
Sep 17 16:30:50.009956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:50.009824    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:50.030182 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:50.030126    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podStartSLOduration=3.194354234 podStartE2EDuration="7.030109997s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835755763s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:30:50.029169367 +0000 UTC m=+2216.294684367" watchObservedRunningTime="2026-09-17 16:30:50.030109997 +0000 UTC m=+2216.295624993"
Sep 17 16:30:51.015459 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:51.015425    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:51.016859 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:51.016824    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:30:52.019754 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:52.019711    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:30:57.024246 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:57.024217    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:30:57.024776 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:30:57.024748    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:31:07.024830 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:31:07.024745    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:31:17.025362 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:31:17.025321    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:31:27.025416 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:31:27.025374    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:31:37.025524 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:31:37.025484    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:31:47.024825 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:31:47.024777    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:31:57.024991 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:31:57.024938    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:32:07.025506 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.025468    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:32:07.089439 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.089399    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"]
Sep 17 16:32:07.092976 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.092955    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.096592 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.096559    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 16:32:07.096741 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.096715    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 16:32:07.107888 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.107860    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"]
Sep 17 16:32:07.273214 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.273170    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.273434 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.273238    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c370f83e-0884-4459-9a44-fe68f790a052-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.273434 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.273264    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.273434 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.273323    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhx2f\" (UniqueName: \"kubernetes.io/projected/c370f83e-0884-4459-9a44-fe68f790a052-kube-api-access-hhx2f\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.273434 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.273340    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c370f83e-0884-4459-9a44-fe68f790a052-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.374501 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.374399    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c370f83e-0884-4459-9a44-fe68f790a052-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.374501 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.374442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.374501 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.374484    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhx2f\" (UniqueName: \"kubernetes.io/projected/c370f83e-0884-4459-9a44-fe68f790a052-kube-api-access-hhx2f\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.374501 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.374501    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c370f83e-0884-4459-9a44-fe68f790a052-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.374853 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.374556    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.375045 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.375017    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c370f83e-0884-4459-9a44-fe68f790a052-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.375189 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.375170    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.375252 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.375200    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.377233 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.377216    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c370f83e-0884-4459-9a44-fe68f790a052-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.386133 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.386108    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhx2f\" (UniqueName: \"kubernetes.io/projected/c370f83e-0884-4459-9a44-fe68f790a052-kube-api-access-hhx2f\") pod \"isvc-xgboost-predictor-7566d65bf7-jjljx\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.404193 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.404159    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:07.683705 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:07.683677    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"]
Sep 17 16:32:07.686207 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:32:07.686177    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc370f83e_0884_4459_9a44_fe68f790a052.slice/crio-a6a7dab3436f48c410ee2dcf5ddd4e477412d419ac916e4bccf78d43732af5e8 WatchSource:0}: Error finding container a6a7dab3436f48c410ee2dcf5ddd4e477412d419ac916e4bccf78d43732af5e8: Status 404 returned error can't find the container with id a6a7dab3436f48c410ee2dcf5ddd4e477412d419ac916e4bccf78d43732af5e8
Sep 17 16:32:08.351657 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:08.351615    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" event={"ID":"c370f83e-0884-4459-9a44-fe68f790a052","Type":"ContainerStarted","Data":"81f7f01ef651b3b59f563d82566f1a0772c35e1e8c54e280be1edfa49673d456"}
Sep 17 16:32:08.352144 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:08.351688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" event={"ID":"c370f83e-0884-4459-9a44-fe68f790a052","Type":"ContainerStarted","Data":"a6a7dab3436f48c410ee2dcf5ddd4e477412d419ac916e4bccf78d43732af5e8"}
Sep 17 16:32:12.371113 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:12.371066    2585 generic.go:356] "Generic (PLEG): container finished" podID="c370f83e-0884-4459-9a44-fe68f790a052" containerID="81f7f01ef651b3b59f563d82566f1a0772c35e1e8c54e280be1edfa49673d456" exitCode=0
Sep 17 16:32:12.371549 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:12.371142    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" event={"ID":"c370f83e-0884-4459-9a44-fe68f790a052","Type":"ContainerDied","Data":"81f7f01ef651b3b59f563d82566f1a0772c35e1e8c54e280be1edfa49673d456"}
Sep 17 16:32:34.483659 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:34.483620    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" event={"ID":"c370f83e-0884-4459-9a44-fe68f790a052","Type":"ContainerStarted","Data":"bd008686aca2fe8e5c2f22b1d35e5643554aa3443a44fc2e2dbb781fd7c4601f"}
Sep 17 16:32:34.484054 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:34.483685    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" event={"ID":"c370f83e-0884-4459-9a44-fe68f790a052","Type":"ContainerStarted","Data":"c5678aa0a74bf1704223612ec563222450dc9e63f652b6813d859308999e9e5d"}
Sep 17 16:32:34.484054 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:34.483974    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:34.484162 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:34.484140    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:34.485426 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:34.485399    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:32:34.507054 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:34.507002    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podStartSLOduration=1.988946932 podStartE2EDuration="27.506963172s" totalImagesPullingTime="21.409511056s" totalInitContainerRuntime="4.108505184s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:32:34.506368952 +0000 UTC m=+2320.771883951" watchObservedRunningTime="2026-09-17 16:32:34.506963172 +0000 UTC m=+2320.772478172"
Sep 17 16:32:35.487949 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:35.487902    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:32:40.492495 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:40.492423    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:32:40.493015 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:40.492989    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:32:50.493440 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:32:50.493395    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:33:00.493487 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:00.493436    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:33:10.493098 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:10.493054    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:33:20.493552 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:20.493507    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:33:30.493047 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:30.493003    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:33:40.493486 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:40.493451    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:33:47.568121 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:47.568084    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"]
Sep 17 16:33:47.568537 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:47.568405    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" containerID="cri-o://c5678aa0a74bf1704223612ec563222450dc9e63f652b6813d859308999e9e5d" gracePeriod=30
Sep 17 16:33:47.568537 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:47.568453    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kube-rbac-proxy" containerID="cri-o://bd008686aca2fe8e5c2f22b1d35e5643554aa3443a44fc2e2dbb781fd7c4601f" gracePeriod=30
Sep 17 16:33:47.804772 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:47.804737    2585 generic.go:356] "Generic (PLEG): container finished" podID="c370f83e-0884-4459-9a44-fe68f790a052" containerID="bd008686aca2fe8e5c2f22b1d35e5643554aa3443a44fc2e2dbb781fd7c4601f" exitCode=2
Sep 17 16:33:47.804947 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:47.804811    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" event={"ID":"c370f83e-0884-4459-9a44-fe68f790a052","Type":"ContainerDied","Data":"bd008686aca2fe8e5c2f22b1d35e5643554aa3443a44fc2e2dbb781fd7c4601f"}
Sep 17 16:33:50.488746 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:50.488695    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 16:33:50.493093 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:50.493061    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:33:51.825829 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:51.825791    2585 generic.go:356] "Generic (PLEG): container finished" podID="c370f83e-0884-4459-9a44-fe68f790a052" containerID="c5678aa0a74bf1704223612ec563222450dc9e63f652b6813d859308999e9e5d" exitCode=0
Sep 17 16:33:51.826194 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:51.825844    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" event={"ID":"c370f83e-0884-4459-9a44-fe68f790a052","Type":"ContainerDied","Data":"c5678aa0a74bf1704223612ec563222450dc9e63f652b6813d859308999e9e5d"}
Sep 17 16:33:51.908924 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:51.908896    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:33:52.019452 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.019417    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c370f83e-0884-4459-9a44-fe68f790a052-proxy-tls\") pod \"c370f83e-0884-4459-9a44-fe68f790a052\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") "
Sep 17 16:33:52.019656 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.019484    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhx2f\" (UniqueName: \"kubernetes.io/projected/c370f83e-0884-4459-9a44-fe68f790a052-kube-api-access-hhx2f\") pod \"c370f83e-0884-4459-9a44-fe68f790a052\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") "
Sep 17 16:33:52.019656 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.019511    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c370f83e-0884-4459-9a44-fe68f790a052-kserve-provision-location\") pod \"c370f83e-0884-4459-9a44-fe68f790a052\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") "
Sep 17 16:33:52.019656 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.019547    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-cabundle-cert\") pod \"c370f83e-0884-4459-9a44-fe68f790a052\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") "
Sep 17 16:33:52.019656 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.019583    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-kube-rbac-proxy-sar-config\") pod \"c370f83e-0884-4459-9a44-fe68f790a052\" (UID: \"c370f83e-0884-4459-9a44-fe68f790a052\") "
Sep 17 16:33:52.019925 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.019896    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c370f83e-0884-4459-9a44-fe68f790a052-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c370f83e-0884-4459-9a44-fe68f790a052" (UID: "c370f83e-0884-4459-9a44-fe68f790a052"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:33:52.019988 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.019936    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c370f83e-0884-4459-9a44-fe68f790a052" (UID: "c370f83e-0884-4459-9a44-fe68f790a052"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:33:52.020056 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.020025    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c370f83e-0884-4459-9a44-fe68f790a052" (UID: "c370f83e-0884-4459-9a44-fe68f790a052"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:33:52.022365 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.022343    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c370f83e-0884-4459-9a44-fe68f790a052-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c370f83e-0884-4459-9a44-fe68f790a052" (UID: "c370f83e-0884-4459-9a44-fe68f790a052"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:33:52.022365 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.022351    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c370f83e-0884-4459-9a44-fe68f790a052-kube-api-access-hhx2f" (OuterVolumeSpecName: "kube-api-access-hhx2f") pod "c370f83e-0884-4459-9a44-fe68f790a052" (UID: "c370f83e-0884-4459-9a44-fe68f790a052"). InnerVolumeSpecName "kube-api-access-hhx2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:33:52.120310 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.120262    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhx2f\" (UniqueName: \"kubernetes.io/projected/c370f83e-0884-4459-9a44-fe68f790a052-kube-api-access-hhx2f\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:33:52.120310 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.120307    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c370f83e-0884-4459-9a44-fe68f790a052-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:33:52.120310 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.120320    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:33:52.120556 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.120331    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c370f83e-0884-4459-9a44-fe68f790a052-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:33:52.120556 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.120340    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c370f83e-0884-4459-9a44-fe68f790a052-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:33:52.833975 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.833938    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx" event={"ID":"c370f83e-0884-4459-9a44-fe68f790a052","Type":"ContainerDied","Data":"a6a7dab3436f48c410ee2dcf5ddd4e477412d419ac916e4bccf78d43732af5e8"}
Sep 17 16:33:52.834452 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.834017    2585 scope.go:117] "RemoveContainer" containerID="bd008686aca2fe8e5c2f22b1d35e5643554aa3443a44fc2e2dbb781fd7c4601f"
Sep 17 16:33:52.834452 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.833961    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"
Sep 17 16:33:52.843033 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.843015    2585 scope.go:117] "RemoveContainer" containerID="c5678aa0a74bf1704223612ec563222450dc9e63f652b6813d859308999e9e5d"
Sep 17 16:33:52.851347 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.851320    2585 scope.go:117] "RemoveContainer" containerID="81f7f01ef651b3b59f563d82566f1a0772c35e1e8c54e280be1edfa49673d456"
Sep 17 16:33:52.884740 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.884712    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"]
Sep 17 16:33:52.903259 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:52.903224    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jjljx"]
Sep 17 16:33:53.783467 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:33:53.783438    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c370f83e-0884-4459-9a44-fe68f790a052" path="/var/lib/kubelet/pods/c370f83e-0884-4459-9a44-fe68f790a052/volumes"
Sep 17 16:34:25.932494 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:25.932460    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:34:25.934994 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:25.934968    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:34:48.944822 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:48.944780    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"]
Sep 17 16:34:48.947509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:48.945092    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container" containerID="cri-o://e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1" gracePeriod=30
Sep 17 16:34:48.947509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:48.945158    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kube-rbac-proxy" containerID="cri-o://8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78" gracePeriod=30
Sep 17 16:34:49.080440 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:49.080409    2585 generic.go:356] "Generic (PLEG): container finished" podID="d4174708-9d4c-4150-aa10-b07235c55a38" containerID="8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78" exitCode=2
Sep 17 16:34:49.080610 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:49.080490    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" event={"ID":"d4174708-9d4c-4150-aa10-b07235c55a38","Type":"ContainerDied","Data":"8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78"}
Sep 17 16:34:52.020673 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:52.020621    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 16:34:53.898094 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.898068    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:34:53.956974 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.956939    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-cabundle-cert\") pod \"d4174708-9d4c-4150-aa10-b07235c55a38\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") "
Sep 17 16:34:53.957184 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.957023    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-kube-rbac-proxy-sar-config\") pod \"d4174708-9d4c-4150-aa10-b07235c55a38\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") "
Sep 17 16:34:53.957184 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.957058    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4174708-9d4c-4150-aa10-b07235c55a38-proxy-tls\") pod \"d4174708-9d4c-4150-aa10-b07235c55a38\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") "
Sep 17 16:34:53.957184 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.957105    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4174708-9d4c-4150-aa10-b07235c55a38-kserve-provision-location\") pod \"d4174708-9d4c-4150-aa10-b07235c55a38\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") "
Sep 17 16:34:53.957184 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.957129    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnn8r\" (UniqueName: \"kubernetes.io/projected/d4174708-9d4c-4150-aa10-b07235c55a38-kube-api-access-qnn8r\") pod \"d4174708-9d4c-4150-aa10-b07235c55a38\" (UID: \"d4174708-9d4c-4150-aa10-b07235c55a38\") "
Sep 17 16:34:53.957460 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.957426    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4174708-9d4c-4150-aa10-b07235c55a38" (UID: "d4174708-9d4c-4150-aa10-b07235c55a38"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:53.957515 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.957429    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4174708-9d4c-4150-aa10-b07235c55a38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4174708-9d4c-4150-aa10-b07235c55a38" (UID: "d4174708-9d4c-4150-aa10-b07235c55a38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:53.957515 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.957431    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4174708-9d4c-4150-aa10-b07235c55a38" (UID: "d4174708-9d4c-4150-aa10-b07235c55a38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:53.959515 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.959477    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4174708-9d4c-4150-aa10-b07235c55a38-kube-api-access-qnn8r" (OuterVolumeSpecName: "kube-api-access-qnn8r") pod "d4174708-9d4c-4150-aa10-b07235c55a38" (UID: "d4174708-9d4c-4150-aa10-b07235c55a38"). InnerVolumeSpecName "kube-api-access-qnn8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:34:53.959657 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:53.959599    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4174708-9d4c-4150-aa10-b07235c55a38-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4174708-9d4c-4150-aa10-b07235c55a38" (UID: "d4174708-9d4c-4150-aa10-b07235c55a38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:34:54.058264 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.058226    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.058264 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.058258    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4174708-9d4c-4150-aa10-b07235c55a38-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.058264 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.058268    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4174708-9d4c-4150-aa10-b07235c55a38-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.058264 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.058277    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnn8r\" (UniqueName: \"kubernetes.io/projected/d4174708-9d4c-4150-aa10-b07235c55a38-kube-api-access-qnn8r\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.058625 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.058305    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4174708-9d4c-4150-aa10-b07235c55a38-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.106915 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.106877    2585 generic.go:356] "Generic (PLEG): container finished" podID="d4174708-9d4c-4150-aa10-b07235c55a38" containerID="e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1" exitCode=0
Sep 17 16:34:54.107131 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.106958    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"
Sep 17 16:34:54.107131 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.106958    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" event={"ID":"d4174708-9d4c-4150-aa10-b07235c55a38","Type":"ContainerDied","Data":"e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1"}
Sep 17 16:34:54.107131 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.107034    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc" event={"ID":"d4174708-9d4c-4150-aa10-b07235c55a38","Type":"ContainerDied","Data":"9f3b71630e2148d5e4357ae87aba559d7ed4a2d5909eebd16cca3ec4b5831ac4"}
Sep 17 16:34:54.107131 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.107076    2585 scope.go:117] "RemoveContainer" containerID="8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78"
Sep 17 16:34:54.118375 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.118353    2585 scope.go:117] "RemoveContainer" containerID="e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1"
Sep 17 16:34:54.130197 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.130171    2585 scope.go:117] "RemoveContainer" containerID="50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536"
Sep 17 16:34:54.135420 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.135394    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"]
Sep 17 16:34:54.140853 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.140829    2585 scope.go:117] "RemoveContainer" containerID="8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78"
Sep 17 16:34:54.141277 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:34:54.141255    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78\": container with ID starting with 8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78 not found: ID does not exist" containerID="8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78"
Sep 17 16:34:54.141387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.141311    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78"} err="failed to get container status \"8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78\": rpc error: code = NotFound desc = could not find container \"8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78\": container with ID starting with 8bcedc0a597c3a56353f10f3390d2ac325e8bd9b84619013b0a6e30f74ed7f78 not found: ID does not exist"
Sep 17 16:34:54.141387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.141339    2585 scope.go:117] "RemoveContainer" containerID="e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1"
Sep 17 16:34:54.141701 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:34:54.141672    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1\": container with ID starting with e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1 not found: ID does not exist" containerID="e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1"
Sep 17 16:34:54.141783 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.141710    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1"} err="failed to get container status \"e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1\": rpc error: code = NotFound desc = could not find container \"e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1\": container with ID starting with e44238b34360f00fc691bbbbe2e247be962474703e5d9a0cede4b12060b16ef1 not found: ID does not exist"
Sep 17 16:34:54.141783 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.141735    2585 scope.go:117] "RemoveContainer" containerID="50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536"
Sep 17 16:34:54.142029 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:34:54.142004    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536\": container with ID starting with 50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536 not found: ID does not exist" containerID="50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536"
Sep 17 16:34:54.142144 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.142034    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536"} err="failed to get container status \"50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536\": rpc error: code = NotFound desc = could not find container \"50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536\": container with ID starting with 50dc9c675501a69b3efde4ed58ad43376a4c3146216179c190babbd845428536 not found: ID does not exist"
Sep 17 16:34:54.150675 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.150646    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-pmcdc"]
Sep 17 16:34:54.242594 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242556    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"]
Sep 17 16:34:54.242941 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242923    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kube-rbac-proxy"
Sep 17 16:34:54.243003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242944    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kube-rbac-proxy"
Sep 17 16:34:54.243003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242958    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kube-rbac-proxy"
Sep 17 16:34:54.243003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242965    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kube-rbac-proxy"
Sep 17 16:34:54.243003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242975    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="storage-initializer"
Sep 17 16:34:54.243003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242981    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="storage-initializer"
Sep 17 16:34:54.243003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242990    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container"
Sep 17 16:34:54.243003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.242995    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container"
Sep 17 16:34:54.243387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.243010    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container"
Sep 17 16:34:54.243387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.243017    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container"
Sep 17 16:34:54.243387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.243029    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="storage-initializer"
Sep 17 16:34:54.243387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.243034    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="storage-initializer"
Sep 17 16:34:54.243387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.243084    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kube-rbac-proxy"
Sep 17 16:34:54.243387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.243094    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kube-rbac-proxy"
Sep 17 16:34:54.243387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.243102    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" containerName="kserve-container"
Sep 17 16:34:54.243387 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.243109    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c370f83e-0884-4459-9a44-fe68f790a052" containerName="kserve-container"
Sep 17 16:34:54.247582 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.247565    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.250537 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.250518    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 16:34:54.250962 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.250944    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 16:34:54.251445 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.251298    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:34:54.251445 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.251316    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:34:54.251445 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.251391    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:34:54.251445 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.251381    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:34:54.251445 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.251424    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:34:54.263493 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.263470    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"]
Sep 17 16:34:54.361403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.361274    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.361403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.361354    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj28v\" (UniqueName: \"kubernetes.io/projected/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-api-access-jj28v\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.361662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.361486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.361662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.361529    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.361662 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.361574    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.463036 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.462993    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.463036 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.463033    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.463352 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.463065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.463352 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.463172    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.463352 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.463218    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj28v\" (UniqueName: \"kubernetes.io/projected/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-api-access-jj28v\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.463650 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.463624    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.463925 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.463903    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.464027 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.464005    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.465879 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.465861    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.476166 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.476139    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj28v\" (UniqueName: \"kubernetes.io/projected/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-api-access-jj28v\") pod \"isvc-canary-stop-predictor-7f68f9d4d-vmxjt\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.558977 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.558938    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:34:54.851990 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:54.851747    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"]
Sep 17 16:34:55.115169 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:55.115052    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" event={"ID":"e90bd1e9-4f32-4ce4-bad1-7c0245852f78","Type":"ContainerStarted","Data":"ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa"}
Sep 17 16:34:55.115606 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:55.115187    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" event={"ID":"e90bd1e9-4f32-4ce4-bad1-7c0245852f78","Type":"ContainerStarted","Data":"4cb2164b9f8452c7b2db61c61c32dc2617d2ae701e153f5d95f98794147f05d1"}
Sep 17 16:34:55.782444 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:55.782407    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4174708-9d4c-4150-aa10-b07235c55a38" path="/var/lib/kubelet/pods/d4174708-9d4c-4150-aa10-b07235c55a38/volumes"
Sep 17 16:34:59.134034 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:59.133938    2585 generic.go:356] "Generic (PLEG): container finished" podID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerID="ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa" exitCode=0
Sep 17 16:34:59.134034 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:34:59.134020    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" event={"ID":"e90bd1e9-4f32-4ce4-bad1-7c0245852f78","Type":"ContainerDied","Data":"ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa"}
Sep 17 16:35:00.146100 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:00.146063    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" event={"ID":"e90bd1e9-4f32-4ce4-bad1-7c0245852f78","Type":"ContainerStarted","Data":"b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29"}
Sep 17 16:35:00.146548 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:00.146151    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" event={"ID":"e90bd1e9-4f32-4ce4-bad1-7c0245852f78","Type":"ContainerStarted","Data":"bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399"}
Sep 17 16:35:00.146548 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:00.146437    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:35:00.146644 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:00.146569    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:35:00.147902 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:00.147871    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:35:00.170783 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:00.170732    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podStartSLOduration=2.219481768 podStartE2EDuration="6.170717897s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.951236129s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:35:00.168995824 +0000 UTC m=+2466.434510821" watchObservedRunningTime="2026-09-17 16:35:00.170717897 +0000 UTC m=+2466.436232897"
Sep 17 16:35:01.151086 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:01.151048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:35:06.156063 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:06.156023    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:35:06.156593 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:06.156567    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:35:16.156535 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:16.156494    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:35:26.156550 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:26.156506    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:35:36.156945 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:36.156898    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:35:43.311390 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.311348    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"]
Sep 17 16:35:43.315270 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.315246    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.317745 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.317724    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 16:35:43.317745 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.317737    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:35:43.326900 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.326872    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"]
Sep 17 16:35:43.399303 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.399255    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hb2s\" (UniqueName: \"kubernetes.io/projected/2db93cf6-0149-41db-9741-72d4730aee92-kube-api-access-9hb2s\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.399509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.399336    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db93cf6-0149-41db-9741-72d4730aee92-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.399509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.399357    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db93cf6-0149-41db-9741-72d4730aee92-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.399509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.399390    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.399509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.399439    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.500392 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.500349    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hb2s\" (UniqueName: \"kubernetes.io/projected/2db93cf6-0149-41db-9741-72d4730aee92-kube-api-access-9hb2s\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.500576 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.500417    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db93cf6-0149-41db-9741-72d4730aee92-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.500576 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.500450    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db93cf6-0149-41db-9741-72d4730aee92-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.500576 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.500508    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.500576 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.500563    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.500965 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.500944    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db93cf6-0149-41db-9741-72d4730aee92-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.501195 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.501176    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.501301 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.501264    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.503023 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.502990    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db93cf6-0149-41db-9741-72d4730aee92-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.508761 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.508740    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hb2s\" (UniqueName: \"kubernetes.io/projected/2db93cf6-0149-41db-9741-72d4730aee92-kube-api-access-9hb2s\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q5tvg\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.628019 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.627916    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:43.910219 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:43.910195    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"]
Sep 17 16:35:43.912359 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:35:43.912327    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db93cf6_0149_41db_9741_72d4730aee92.slice/crio-e6ce0a3701e5ca13c5b01639e8d3c9490d4ba04da8bc24b9ab520f8b3c96fb7b WatchSource:0}: Error finding container e6ce0a3701e5ca13c5b01639e8d3c9490d4ba04da8bc24b9ab520f8b3c96fb7b: Status 404 returned error can't find the container with id e6ce0a3701e5ca13c5b01639e8d3c9490d4ba04da8bc24b9ab520f8b3c96fb7b
Sep 17 16:35:44.345082 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:44.345041    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" event={"ID":"2db93cf6-0149-41db-9741-72d4730aee92","Type":"ContainerStarted","Data":"37e49d90492eee7cfacfa6d06f58b720ed7085e0317e75ddb3a36720ecf4981d"}
Sep 17 16:35:44.345495 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:44.345106    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" event={"ID":"2db93cf6-0149-41db-9741-72d4730aee92","Type":"ContainerStarted","Data":"e6ce0a3701e5ca13c5b01639e8d3c9490d4ba04da8bc24b9ab520f8b3c96fb7b"}
Sep 17 16:35:46.156536 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:46.156496    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:35:48.365096 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:48.365060    2585 generic.go:356] "Generic (PLEG): container finished" podID="2db93cf6-0149-41db-9741-72d4730aee92" containerID="37e49d90492eee7cfacfa6d06f58b720ed7085e0317e75ddb3a36720ecf4981d" exitCode=0
Sep 17 16:35:48.365506 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:48.365134    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" event={"ID":"2db93cf6-0149-41db-9741-72d4730aee92","Type":"ContainerDied","Data":"37e49d90492eee7cfacfa6d06f58b720ed7085e0317e75ddb3a36720ecf4981d"}
Sep 17 16:35:49.371683 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:49.371642    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" event={"ID":"2db93cf6-0149-41db-9741-72d4730aee92","Type":"ContainerStarted","Data":"ea2fdd256f3b5cc6f2a20cc2c0cd1b98fabb13e4858e960a2c528ba3047892cc"}
Sep 17 16:35:49.372136 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:49.371705    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" event={"ID":"2db93cf6-0149-41db-9741-72d4730aee92","Type":"ContainerStarted","Data":"a262368f586b8a1eccfa100e8d79065b5f915fab24ee85b1b8d800d81166d3b3"}
Sep 17 16:35:49.372136 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:49.371984    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:49.372136 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:49.372125    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:49.373534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:49.373507    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:35:49.393073 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:49.393018    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podStartSLOduration=2.478363169 podStartE2EDuration="6.393001464s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.914638295s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:35:49.39115128 +0000 UTC m=+2515.656666283" watchObservedRunningTime="2026-09-17 16:35:49.393001464 +0000 UTC m=+2515.658516463"
Sep 17 16:35:50.376684 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:50.376643    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:35:55.381697 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:55.381669    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:35:55.382201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:55.382173    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:35:56.156517 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:35:56.156476    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:36:05.382615 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:05.382577    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:36:06.156943 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:06.156901    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:36:15.382402 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:15.382356    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:36:16.157998 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:16.157965    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:36:24.180692 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:24.180649    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"]
Sep 17 16:36:24.181094 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:24.180928    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" containerID="cri-o://bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399" gracePeriod=30
Sep 17 16:36:24.181094 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:24.180969    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kube-rbac-proxy" containerID="cri-o://b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29" gracePeriod=30
Sep 17 16:36:24.534437 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:24.534405    2585 generic.go:356] "Generic (PLEG): container finished" podID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerID="b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29" exitCode=2
Sep 17 16:36:24.534605 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:24.534475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" event={"ID":"e90bd1e9-4f32-4ce4-bad1-7c0245852f78","Type":"ContainerDied","Data":"b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29"}
Sep 17 16:36:25.383124 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:25.383085    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:36:26.151865 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:26.151824    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 17 16:36:26.157175 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:26.157152    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:36:28.925050 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:28.925026    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:36:29.021469 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.021441    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-proxy-tls\") pod \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") "
Sep 17 16:36:29.021670 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.021522    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj28v\" (UniqueName: \"kubernetes.io/projected/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-api-access-jj28v\") pod \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") "
Sep 17 16:36:29.021670 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.021553    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kserve-provision-location\") pod \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") "
Sep 17 16:36:29.021794 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.021684    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-rbac-proxy-sar-config\") pod \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") "
Sep 17 16:36:29.021794 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.021743    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-cabundle-cert\") pod \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\" (UID: \"e90bd1e9-4f32-4ce4-bad1-7c0245852f78\") "
Sep 17 16:36:29.022190 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.022133    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e90bd1e9-4f32-4ce4-bad1-7c0245852f78" (UID: "e90bd1e9-4f32-4ce4-bad1-7c0245852f78"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:36:29.022340 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.022210    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e90bd1e9-4f32-4ce4-bad1-7c0245852f78" (UID: "e90bd1e9-4f32-4ce4-bad1-7c0245852f78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:36:29.022340 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.022256    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e90bd1e9-4f32-4ce4-bad1-7c0245852f78" (UID: "e90bd1e9-4f32-4ce4-bad1-7c0245852f78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:36:29.024015 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.023993    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e90bd1e9-4f32-4ce4-bad1-7c0245852f78" (UID: "e90bd1e9-4f32-4ce4-bad1-7c0245852f78"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:36:29.024179 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.024161    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-api-access-jj28v" (OuterVolumeSpecName: "kube-api-access-jj28v") pod "e90bd1e9-4f32-4ce4-bad1-7c0245852f78" (UID: "e90bd1e9-4f32-4ce4-bad1-7c0245852f78"). InnerVolumeSpecName "kube-api-access-jj28v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:36:29.122991 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.122957    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj28v\" (UniqueName: \"kubernetes.io/projected/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-api-access-jj28v\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.122991 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.122987    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.122991 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.122997    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.123239 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.123006    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.123239 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.123016    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90bd1e9-4f32-4ce4-bad1-7c0245852f78-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.560788 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.560746    2585 generic.go:356] "Generic (PLEG): container finished" podID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerID="bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399" exitCode=0
Sep 17 16:36:29.560961 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.560811    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"
Sep 17 16:36:29.560961 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.560810    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" event={"ID":"e90bd1e9-4f32-4ce4-bad1-7c0245852f78","Type":"ContainerDied","Data":"bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399"}
Sep 17 16:36:29.560961 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.560942    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt" event={"ID":"e90bd1e9-4f32-4ce4-bad1-7c0245852f78","Type":"ContainerDied","Data":"4cb2164b9f8452c7b2db61c61c32dc2617d2ae701e153f5d95f98794147f05d1"}
Sep 17 16:36:29.561065 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.560970    2585 scope.go:117] "RemoveContainer" containerID="b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29"
Sep 17 16:36:29.570174 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.570153    2585 scope.go:117] "RemoveContainer" containerID="bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399"
Sep 17 16:36:29.579621 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.579594    2585 scope.go:117] "RemoveContainer" containerID="ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa"
Sep 17 16:36:29.584563 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.584530    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"]
Sep 17 16:36:29.586411 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.586389    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-vmxjt"]
Sep 17 16:36:29.588692 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.588670    2585 scope.go:117] "RemoveContainer" containerID="b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29"
Sep 17 16:36:29.589017 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:36:29.588991    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29\": container with ID starting with b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29 not found: ID does not exist" containerID="b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29"
Sep 17 16:36:29.589071 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.589028    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29"} err="failed to get container status \"b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29\": rpc error: code = NotFound desc = could not find container \"b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29\": container with ID starting with b4e3eb60df5ba0ccda3052ec12f09bec1e5edde544acc0448d0d7eff404fff29 not found: ID does not exist"
Sep 17 16:36:29.589071 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.589047    2585 scope.go:117] "RemoveContainer" containerID="bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399"
Sep 17 16:36:29.589360 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:36:29.589342    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399\": container with ID starting with bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399 not found: ID does not exist" containerID="bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399"
Sep 17 16:36:29.589411 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.589367    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399"} err="failed to get container status \"bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399\": rpc error: code = NotFound desc = could not find container \"bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399\": container with ID starting with bd7a67830563c88e2839dd046b9fc2f4bda823a6619fa36d21afeec789863399 not found: ID does not exist"
Sep 17 16:36:29.589411 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.589383    2585 scope.go:117] "RemoveContainer" containerID="ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa"
Sep 17 16:36:29.589616 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:36:29.589599    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa\": container with ID starting with ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa not found: ID does not exist" containerID="ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa"
Sep 17 16:36:29.589673 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.589620    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa"} err="failed to get container status \"ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa\": rpc error: code = NotFound desc = could not find container \"ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa\": container with ID starting with ff0ea04b06b198ca0e33dc38b53af48e3ff1277d2ba34747e13bc8c8facda5fa not found: ID does not exist"
Sep 17 16:36:29.783174 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:29.783139    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" path="/var/lib/kubelet/pods/e90bd1e9-4f32-4ce4-bad1-7c0245852f78/volumes"
Sep 17 16:36:35.382241 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:35.382193    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:36:45.382545 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:45.382505    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:36:55.382365 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:36:55.382311    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:37:05.383471 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:05.383435    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:37:13.897053 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:13.896968    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"]
Sep 17 16:37:13.897453 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:13.897216    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" containerID="cri-o://a262368f586b8a1eccfa100e8d79065b5f915fab24ee85b1b8d800d81166d3b3" gracePeriod=30
Sep 17 16:37:13.897453 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:13.897236    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kube-rbac-proxy" containerID="cri-o://ea2fdd256f3b5cc6f2a20cc2c0cd1b98fabb13e4858e960a2c528ba3047892cc" gracePeriod=30
Sep 17 16:37:14.765712 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:14.765682    2585 generic.go:356] "Generic (PLEG): container finished" podID="2db93cf6-0149-41db-9741-72d4730aee92" containerID="ea2fdd256f3b5cc6f2a20cc2c0cd1b98fabb13e4858e960a2c528ba3047892cc" exitCode=2
Sep 17 16:37:14.765891 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:14.765748    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" event={"ID":"2db93cf6-0149-41db-9741-72d4730aee92","Type":"ContainerDied","Data":"ea2fdd256f3b5cc6f2a20cc2c0cd1b98fabb13e4858e960a2c528ba3047892cc"}
Sep 17 16:37:15.376908 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:15.376866    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 17 16:37:15.382219 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:15.382195    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:37:17.783378 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.783335    2585 generic.go:356] "Generic (PLEG): container finished" podID="2db93cf6-0149-41db-9741-72d4730aee92" containerID="a262368f586b8a1eccfa100e8d79065b5f915fab24ee85b1b8d800d81166d3b3" exitCode=0
Sep 17 16:37:17.784089 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.784053    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" event={"ID":"2db93cf6-0149-41db-9741-72d4730aee92","Type":"ContainerDied","Data":"a262368f586b8a1eccfa100e8d79065b5f915fab24ee85b1b8d800d81166d3b3"}
Sep 17 16:37:17.844031 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.844001    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:37:17.856833 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.856811    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db93cf6-0149-41db-9741-72d4730aee92-kserve-provision-location\") pod \"2db93cf6-0149-41db-9741-72d4730aee92\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") "
Sep 17 16:37:17.856938 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.856869    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hb2s\" (UniqueName: \"kubernetes.io/projected/2db93cf6-0149-41db-9741-72d4730aee92-kube-api-access-9hb2s\") pod \"2db93cf6-0149-41db-9741-72d4730aee92\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") "
Sep 17 16:37:17.856938 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.856899    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db93cf6-0149-41db-9741-72d4730aee92-proxy-tls\") pod \"2db93cf6-0149-41db-9741-72d4730aee92\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") "
Sep 17 16:37:17.857048 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.856941    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-kube-rbac-proxy-sar-config\") pod \"2db93cf6-0149-41db-9741-72d4730aee92\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") "
Sep 17 16:37:17.857222 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.857149    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db93cf6-0149-41db-9741-72d4730aee92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2db93cf6-0149-41db-9741-72d4730aee92" (UID: "2db93cf6-0149-41db-9741-72d4730aee92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:37:17.857222 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.857159    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-cabundle-cert\") pod \"2db93cf6-0149-41db-9741-72d4730aee92\" (UID: \"2db93cf6-0149-41db-9741-72d4730aee92\") "
Sep 17 16:37:17.857370 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.857246    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2db93cf6-0149-41db-9741-72d4730aee92" (UID: "2db93cf6-0149-41db-9741-72d4730aee92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:37:17.857520 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.857505    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:37:17.857567 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.857514    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2db93cf6-0149-41db-9741-72d4730aee92" (UID: "2db93cf6-0149-41db-9741-72d4730aee92"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:37:17.857567 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.857523    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db93cf6-0149-41db-9741-72d4730aee92-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:37:17.859170 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.859148    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db93cf6-0149-41db-9741-72d4730aee92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2db93cf6-0149-41db-9741-72d4730aee92" (UID: "2db93cf6-0149-41db-9741-72d4730aee92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:37:17.859237 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.859199    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db93cf6-0149-41db-9741-72d4730aee92-kube-api-access-9hb2s" (OuterVolumeSpecName: "kube-api-access-9hb2s") pod "2db93cf6-0149-41db-9741-72d4730aee92" (UID: "2db93cf6-0149-41db-9741-72d4730aee92"). InnerVolumeSpecName "kube-api-access-9hb2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:37:17.958496 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.958463    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hb2s\" (UniqueName: \"kubernetes.io/projected/2db93cf6-0149-41db-9741-72d4730aee92-kube-api-access-9hb2s\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:37:17.958496 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.958489    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db93cf6-0149-41db-9741-72d4730aee92-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:37:17.958496 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:17.958499    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2db93cf6-0149-41db-9741-72d4730aee92-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:37:18.789207 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:18.789166    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg" event={"ID":"2db93cf6-0149-41db-9741-72d4730aee92","Type":"ContainerDied","Data":"e6ce0a3701e5ca13c5b01639e8d3c9490d4ba04da8bc24b9ab520f8b3c96fb7b"}
Sep 17 16:37:18.789207 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:18.789192    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"
Sep 17 16:37:18.789708 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:18.789233    2585 scope.go:117] "RemoveContainer" containerID="ea2fdd256f3b5cc6f2a20cc2c0cd1b98fabb13e4858e960a2c528ba3047892cc"
Sep 17 16:37:18.798335 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:18.798314    2585 scope.go:117] "RemoveContainer" containerID="a262368f586b8a1eccfa100e8d79065b5f915fab24ee85b1b8d800d81166d3b3"
Sep 17 16:37:18.806274 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:18.806252    2585 scope.go:117] "RemoveContainer" containerID="37e49d90492eee7cfacfa6d06f58b720ed7085e0317e75ddb3a36720ecf4981d"
Sep 17 16:37:18.813643 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:18.813621    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"]
Sep 17 16:37:18.820372 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:18.820343    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q5tvg"]
Sep 17 16:37:19.782691 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:37:19.782660    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db93cf6-0149-41db-9741-72d4730aee92" path="/var/lib/kubelet/pods/2db93cf6-0149-41db-9741-72d4730aee92/volumes"
Sep 17 16:38:24.367919 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.367878    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"]
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368225    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kube-rbac-proxy"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368238    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kube-rbac-proxy"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368251    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="storage-initializer"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368257    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="storage-initializer"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368263    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368269    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368277    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368307    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368315    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kube-rbac-proxy"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368321    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kube-rbac-proxy"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368342    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="storage-initializer"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368347    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="storage-initializer"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368396    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kserve-container"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368404    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kserve-container"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368409    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90bd1e9-4f32-4ce4-bad1-7c0245852f78" containerName="kube-rbac-proxy"
Sep 17 16:38:24.368491 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.368419    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db93cf6-0149-41db-9741-72d4730aee92" containerName="kube-rbac-proxy"
Sep 17 16:38:24.371689 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.371670    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.374470 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.374448    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:38:24.374595 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.374456    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:38:24.374801 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.374784    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:38:24.375543 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.375524    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:38:24.375612 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.375551    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:38:24.375612 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.375528    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:38:24.375612 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.375569    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 16:38:24.387322 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.387275    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"]
Sep 17 16:38:24.418665 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.418629    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/152f85ad-502a-4ba5-92d1-3f049a8d63e9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.418665 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.418668    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.418912 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.418705    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.418912 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.418850    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.418912 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.418901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6krv\" (UniqueName: \"kubernetes.io/projected/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-api-access-b6krv\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.520125 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.520091    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/152f85ad-502a-4ba5-92d1-3f049a8d63e9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.520125 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.520129    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.520379 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.520172    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.520379 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.520230    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.520379 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.520250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6krv\" (UniqueName: \"kubernetes.io/projected/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-api-access-b6krv\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.520722 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.520699    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.520917 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.520897    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.521019 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.521002    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.522796 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.522777    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/152f85ad-502a-4ba5-92d1-3f049a8d63e9-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.530382 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.530357    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6krv\" (UniqueName: \"kubernetes.io/projected/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-api-access-b6krv\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qxl2w\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.682277 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.682181    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:24.976262 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:24.976234    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"]
Sep 17 16:38:24.978340 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:38:24.978313    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152f85ad_502a_4ba5_92d1_3f049a8d63e9.slice/crio-20af574d1cd02bcca0f087ecf17a931fe06bbef9d50636fce4f8b0179a2a6aa6 WatchSource:0}: Error finding container 20af574d1cd02bcca0f087ecf17a931fe06bbef9d50636fce4f8b0179a2a6aa6: Status 404 returned error can't find the container with id 20af574d1cd02bcca0f087ecf17a931fe06bbef9d50636fce4f8b0179a2a6aa6
Sep 17 16:38:25.073347 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:25.073312    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" event={"ID":"152f85ad-502a-4ba5-92d1-3f049a8d63e9","Type":"ContainerStarted","Data":"d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940"}
Sep 17 16:38:25.073509 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:25.073374    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" event={"ID":"152f85ad-502a-4ba5-92d1-3f049a8d63e9","Type":"ContainerStarted","Data":"20af574d1cd02bcca0f087ecf17a931fe06bbef9d50636fce4f8b0179a2a6aa6"}
Sep 17 16:38:29.091936 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:29.091840    2585 generic.go:356] "Generic (PLEG): container finished" podID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerID="d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940" exitCode=0
Sep 17 16:38:29.092360 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:29.091923    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" event={"ID":"152f85ad-502a-4ba5-92d1-3f049a8d63e9","Type":"ContainerDied","Data":"d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940"}
Sep 17 16:38:30.098210 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:30.098174    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" event={"ID":"152f85ad-502a-4ba5-92d1-3f049a8d63e9","Type":"ContainerStarted","Data":"b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b"}
Sep 17 16:38:30.098713 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:30.098248    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" event={"ID":"152f85ad-502a-4ba5-92d1-3f049a8d63e9","Type":"ContainerStarted","Data":"eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde"}
Sep 17 16:38:30.098713 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:30.098559    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:30.098713 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:30.098614    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:30.100191 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:30.100164    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:38:30.120555 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:30.120492    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podStartSLOduration=2.365943448 podStartE2EDuration="6.120474603s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.754531155s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:38:30.118994383 +0000 UTC m=+2676.384509384" watchObservedRunningTime="2026-09-17 16:38:30.120474603 +0000 UTC m=+2676.385989603"
Sep 17 16:38:31.102752 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:31.102712    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:38:36.107880 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:36.107842    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:38:36.108497 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:36.108467    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:38:46.108560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:46.108516    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:38:56.113355 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:38:56.113305    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:39:06.109120 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:06.109078    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:39:16.108567 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:16.108514    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:39:25.964353 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:25.964323    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:39:25.965428 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:25.965409    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:39:26.108795 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:26.108754    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:39:36.109488 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:36.109448    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:39:44.995709 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:44.995664    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"]
Sep 17 16:39:44.996201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:44.995932    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" containerID="cri-o://eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde" gracePeriod=30
Sep 17 16:39:44.996201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:44.995992    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kube-rbac-proxy" containerID="cri-o://b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b" gracePeriod=30
Sep 17 16:39:45.429463 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:45.429427    2585 generic.go:356] "Generic (PLEG): container finished" podID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerID="b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b" exitCode=2
Sep 17 16:39:45.429643 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:45.429503    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" event={"ID":"152f85ad-502a-4ba5-92d1-3f049a8d63e9","Type":"ContainerDied","Data":"b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b"}
Sep 17 16:39:46.103827 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:46.103790    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 17 16:39:46.109325 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:46.109273    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 16:39:49.348340 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.348310    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:39:49.448399 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.448274    2585 generic.go:356] "Generic (PLEG): container finished" podID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerID="eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde" exitCode=0
Sep 17 16:39:49.448399 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.448326    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" event={"ID":"152f85ad-502a-4ba5-92d1-3f049a8d63e9","Type":"ContainerDied","Data":"eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde"}
Sep 17 16:39:49.448399 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.448370    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"
Sep 17 16:39:49.448698 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.448407    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w" event={"ID":"152f85ad-502a-4ba5-92d1-3f049a8d63e9","Type":"ContainerDied","Data":"20af574d1cd02bcca0f087ecf17a931fe06bbef9d50636fce4f8b0179a2a6aa6"}
Sep 17 16:39:49.448698 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.448447    2585 scope.go:117] "RemoveContainer" containerID="b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b"
Sep 17 16:39:49.455533 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.455510    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-rbac-proxy-sar-config\") pod \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") "
Sep 17 16:39:49.455653 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.455566    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kserve-provision-location\") pod \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") "
Sep 17 16:39:49.455653 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.455591    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6krv\" (UniqueName: \"kubernetes.io/projected/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-api-access-b6krv\") pod \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") "
Sep 17 16:39:49.455653 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.455625    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/152f85ad-502a-4ba5-92d1-3f049a8d63e9-proxy-tls\") pod \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") "
Sep 17 16:39:49.455810 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.455652    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-cabundle-cert\") pod \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\" (UID: \"152f85ad-502a-4ba5-92d1-3f049a8d63e9\") "
Sep 17 16:39:49.455905 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.455882    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "152f85ad-502a-4ba5-92d1-3f049a8d63e9" (UID: "152f85ad-502a-4ba5-92d1-3f049a8d63e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:39:49.456044 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.456024    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "152f85ad-502a-4ba5-92d1-3f049a8d63e9" (UID: "152f85ad-502a-4ba5-92d1-3f049a8d63e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:49.456154 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.456135    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "152f85ad-502a-4ba5-92d1-3f049a8d63e9" (UID: "152f85ad-502a-4ba5-92d1-3f049a8d63e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:39:49.457226 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.457201    2585 scope.go:117] "RemoveContainer" containerID="eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde"
Sep 17 16:39:49.458134 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.458108    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-api-access-b6krv" (OuterVolumeSpecName: "kube-api-access-b6krv") pod "152f85ad-502a-4ba5-92d1-3f049a8d63e9" (UID: "152f85ad-502a-4ba5-92d1-3f049a8d63e9"). InnerVolumeSpecName "kube-api-access-b6krv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:39:49.458134 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.458119    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152f85ad-502a-4ba5-92d1-3f049a8d63e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "152f85ad-502a-4ba5-92d1-3f049a8d63e9" (UID: "152f85ad-502a-4ba5-92d1-3f049a8d63e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:39:49.473200 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.473178    2585 scope.go:117] "RemoveContainer" containerID="d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940"
Sep 17 16:39:49.481298 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.481265    2585 scope.go:117] "RemoveContainer" containerID="b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b"
Sep 17 16:39:49.481611 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:39:49.481589    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b\": container with ID starting with b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b not found: ID does not exist" containerID="b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b"
Sep 17 16:39:49.481696 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.481625    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b"} err="failed to get container status \"b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b\": rpc error: code = NotFound desc = could not find container \"b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b\": container with ID starting with b1282f37479a4d58e80cf6ad213991ed2543f2d4791c55fa087cc8ad6e94c16b not found: ID does not exist"
Sep 17 16:39:49.481696 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.481649    2585 scope.go:117] "RemoveContainer" containerID="eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde"
Sep 17 16:39:49.481930 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:39:49.481914    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde\": container with ID starting with eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde not found: ID does not exist" containerID="eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde"
Sep 17 16:39:49.481994 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.481939    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde"} err="failed to get container status \"eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde\": rpc error: code = NotFound desc = could not find container \"eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde\": container with ID starting with eb48bb881ef9c60ef484b589f405c26e2506eaf3dc883ef3caf24aa09dc9cdde not found: ID does not exist"
Sep 17 16:39:49.481994 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.481959    2585 scope.go:117] "RemoveContainer" containerID="d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940"
Sep 17 16:39:49.482204 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:39:49.482186    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940\": container with ID starting with d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940 not found: ID does not exist" containerID="d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940"
Sep 17 16:39:49.482260 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.482210    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940"} err="failed to get container status \"d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940\": rpc error: code = NotFound desc = could not find container \"d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940\": container with ID starting with d0192879e1e6feb3b87dcbc0fe3c23e38df501f9711b87ac771b128af8d45940 not found: ID does not exist"
Sep 17 16:39:49.556808 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.556766    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/152f85ad-502a-4ba5-92d1-3f049a8d63e9-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:39:49.556808 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.556803    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:39:49.557009 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.556816    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:39:49.557009 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.556832    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:39:49.557009 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.556841    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6krv\" (UniqueName: \"kubernetes.io/projected/152f85ad-502a-4ba5-92d1-3f049a8d63e9-kube-api-access-b6krv\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:39:49.773939 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.773905    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"]
Sep 17 16:39:49.785842 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:49.785794    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qxl2w"]
Sep 17 16:39:51.782544 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:39:51.782512    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" path="/var/lib/kubelet/pods/152f85ad-502a-4ba5-92d1-3f049a8d63e9/volumes"
Sep 17 16:41:22.874715 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.874675    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"]
Sep 17 16:41:22.875350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.875047    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container"
Sep 17 16:41:22.875350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.875058    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container"
Sep 17 16:41:22.875350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.875072    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="storage-initializer"
Sep 17 16:41:22.875350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.875078    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="storage-initializer"
Sep 17 16:41:22.875350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.875098    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kube-rbac-proxy"
Sep 17 16:41:22.875350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.875103    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kube-rbac-proxy"
Sep 17 16:41:22.875350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.875161    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kserve-container"
Sep 17 16:41:22.875350 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.875175    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="152f85ad-502a-4ba5-92d1-3f049a8d63e9" containerName="kube-rbac-proxy"
Sep 17 16:41:22.878489 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.878469    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:22.894841 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.894800    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 16:41:22.896576 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.896554    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:41:22.897127 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.897103    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:41:22.897211 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.897102    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 16:41:22.897647 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.897631    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:41:22.897706 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.897635    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:41:22.912597 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.912559    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:41:22.918813 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.918773    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"]
Sep 17 16:41:22.955637 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.955601    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92kk\" (UniqueName: \"kubernetes.io/projected/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-api-access-v92kk\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:22.955637 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.955639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fbd696-8cb1-49c0-a7d4-4899db88130a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:22.955887 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.955661    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:22.955887 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.955750    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:22.955887 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:22.955798    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.056681 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.056634    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.056897 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.056719    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92kk\" (UniqueName: \"kubernetes.io/projected/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-api-access-v92kk\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.056897 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.056754    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fbd696-8cb1-49c0-a7d4-4899db88130a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.056897 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.056785    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.056897 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.056838    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.057201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.057178    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.057446 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.057420    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.057561 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.057516    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.059351 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.059332    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fbd696-8cb1-49c0-a7d4-4899db88130a-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.073196 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.073164    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92kk\" (UniqueName: \"kubernetes.io/projected/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-api-access-v92kk\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.188559 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.188461    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:23.467526 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.467488    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"]
Sep 17 16:41:23.470245 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:41:23.470208    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fbd696_8cb1_49c0_a7d4_4899db88130a.slice/crio-39fb71fd74587c5303d9f81ad2713fbce096f79d44b97256a4aa2aa59e92cdd1 WatchSource:0}: Error finding container 39fb71fd74587c5303d9f81ad2713fbce096f79d44b97256a4aa2aa59e92cdd1: Status 404 returned error can't find the container with id 39fb71fd74587c5303d9f81ad2713fbce096f79d44b97256a4aa2aa59e92cdd1
Sep 17 16:41:23.851023 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.850983    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" event={"ID":"e0fbd696-8cb1-49c0-a7d4-4899db88130a","Type":"ContainerStarted","Data":"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc"}
Sep 17 16:41:23.851206 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:23.851048    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" event={"ID":"e0fbd696-8cb1-49c0-a7d4-4899db88130a","Type":"ContainerStarted","Data":"39fb71fd74587c5303d9f81ad2713fbce096f79d44b97256a4aa2aa59e92cdd1"}
Sep 17 16:41:27.877132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:27.877097    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerID="37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc" exitCode=0
Sep 17 16:41:27.877132 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:27.877140    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" event={"ID":"e0fbd696-8cb1-49c0-a7d4-4899db88130a","Type":"ContainerDied","Data":"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc"}
Sep 17 16:41:27.878401 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:27.878384    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:41:34.916727 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:34.916690    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" event={"ID":"e0fbd696-8cb1-49c0-a7d4-4899db88130a","Type":"ContainerStarted","Data":"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a"}
Sep 17 16:41:34.917176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:34.916781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" event={"ID":"e0fbd696-8cb1-49c0-a7d4-4899db88130a","Type":"ContainerStarted","Data":"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d"}
Sep 17 16:41:34.917176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:34.917043    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:34.917176 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:34.917097    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:34.918637 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:34.918609    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:41:34.940351 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:34.940299    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" podStartSLOduration=2.579242521 podStartE2EDuration="12.940266033s" totalImagesPullingTime="6.551961702s" totalInitContainerRuntime="3.80906181s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:34.938261013 +0000 UTC m=+2861.203776014" watchObservedRunningTime="2026-09-17 16:41:34.940266033 +0000 UTC m=+2861.205781033"
Sep 17 16:41:35.921159 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:35.921117    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:41:40.926175 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:40.926101    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:40.926681 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:40.926655    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:41:50.928142 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:50.928112    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:52.975003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:52.974968    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"]
Sep 17 16:41:52.975465 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:52.975207    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kserve-container" containerID="cri-o://cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d" gracePeriod=30
Sep 17 16:41:52.975465 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:52.975265    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kube-rbac-proxy" containerID="cri-o://2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a" gracePeriod=30
Sep 17 16:41:53.827735 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.827703    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:53.916700 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.916594    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fbd696-8cb1-49c0-a7d4-4899db88130a-proxy-tls\") pod \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") "
Sep 17 16:41:53.916700 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.916637    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-cabundle-cert\") pod \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") "
Sep 17 16:41:53.916700 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.916675    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kserve-provision-location\") pod \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") "
Sep 17 16:41:53.916700 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.916701    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-rbac-proxy-sar-config\") pod \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") "
Sep 17 16:41:53.917032 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.916797    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v92kk\" (UniqueName: \"kubernetes.io/projected/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-api-access-v92kk\") pod \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\" (UID: \"e0fbd696-8cb1-49c0-a7d4-4899db88130a\") "
Sep 17 16:41:53.917128 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.917082    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0fbd696-8cb1-49c0-a7d4-4899db88130a" (UID: "e0fbd696-8cb1-49c0-a7d4-4899db88130a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:53.917182 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.917082    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0fbd696-8cb1-49c0-a7d4-4899db88130a" (UID: "e0fbd696-8cb1-49c0-a7d4-4899db88130a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:41:53.917182 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.917095    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0fbd696-8cb1-49c0-a7d4-4899db88130a" (UID: "e0fbd696-8cb1-49c0-a7d4-4899db88130a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:41:53.919083 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.919061    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-api-access-v92kk" (OuterVolumeSpecName: "kube-api-access-v92kk") pod "e0fbd696-8cb1-49c0-a7d4-4899db88130a" (UID: "e0fbd696-8cb1-49c0-a7d4-4899db88130a"). InnerVolumeSpecName "kube-api-access-v92kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:41:53.919189 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:53.919061    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0fbd696-8cb1-49c0-a7d4-4899db88130a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0fbd696-8cb1-49c0-a7d4-4899db88130a" (UID: "e0fbd696-8cb1-49c0-a7d4-4899db88130a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:41:54.002161 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.002121    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerID="2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a" exitCode=2
Sep 17 16:41:54.002161 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.002150    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerID="cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d" exitCode=0
Sep 17 16:41:54.002676 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.002183    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"
Sep 17 16:41:54.002676 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.002206    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" event={"ID":"e0fbd696-8cb1-49c0-a7d4-4899db88130a","Type":"ContainerDied","Data":"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a"}
Sep 17 16:41:54.002676 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.002266    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" event={"ID":"e0fbd696-8cb1-49c0-a7d4-4899db88130a","Type":"ContainerDied","Data":"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d"}
Sep 17 16:41:54.002676 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.002310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj" event={"ID":"e0fbd696-8cb1-49c0-a7d4-4899db88130a","Type":"ContainerDied","Data":"39fb71fd74587c5303d9f81ad2713fbce096f79d44b97256a4aa2aa59e92cdd1"}
Sep 17 16:41:54.002676 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.002313    2585 scope.go:117] "RemoveContainer" containerID="2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a"
Sep 17 16:41:54.011513 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.011494    2585 scope.go:117] "RemoveContainer" containerID="cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d"
Sep 17 16:41:54.018060 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.018034    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fbd696-8cb1-49c0-a7d4-4899db88130a-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:41:54.018060 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.018061    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:41:54.018222 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.018074    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:41:54.018222 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.018087    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:41:54.018222 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.018100    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v92kk\" (UniqueName: \"kubernetes.io/projected/e0fbd696-8cb1-49c0-a7d4-4899db88130a-kube-api-access-v92kk\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:41:54.022299 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.022259    2585 scope.go:117] "RemoveContainer" containerID="37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc"
Sep 17 16:41:54.030012 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.029980    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"]
Sep 17 16:41:54.033697 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.033669    2585 scope.go:117] "RemoveContainer" containerID="2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a"
Sep 17 16:41:54.033830 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.033672    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4b9jj"]
Sep 17 16:41:54.034067 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:41:54.034047    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a\": container with ID starting with 2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a not found: ID does not exist" containerID="2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a"
Sep 17 16:41:54.034119 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.034077    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a"} err="failed to get container status \"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a\": rpc error: code = NotFound desc = could not find container \"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a\": container with ID starting with 2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a not found: ID does not exist"
Sep 17 16:41:54.034119 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.034096    2585 scope.go:117] "RemoveContainer" containerID="cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d"
Sep 17 16:41:54.034449 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:41:54.034426    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d\": container with ID starting with cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d not found: ID does not exist" containerID="cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d"
Sep 17 16:41:54.034548 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.034455    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d"} err="failed to get container status \"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d\": rpc error: code = NotFound desc = could not find container \"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d\": container with ID starting with cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d not found: ID does not exist"
Sep 17 16:41:54.034548 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.034475    2585 scope.go:117] "RemoveContainer" containerID="37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc"
Sep 17 16:41:54.034938 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:41:54.034915    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc\": container with ID starting with 37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc not found: ID does not exist" containerID="37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc"
Sep 17 16:41:54.035031 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.034942    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc"} err="failed to get container status \"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc\": rpc error: code = NotFound desc = could not find container \"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc\": container with ID starting with 37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc not found: ID does not exist"
Sep 17 16:41:54.035031 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.034961    2585 scope.go:117] "RemoveContainer" containerID="2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a"
Sep 17 16:41:54.035238 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.035221    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a"} err="failed to get container status \"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a\": rpc error: code = NotFound desc = could not find container \"2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a\": container with ID starting with 2a00b284ba958df93799fed671f0f3f9b5aa99fc5fd360e93e4775656f50e92a not found: ID does not exist"
Sep 17 16:41:54.035319 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.035239    2585 scope.go:117] "RemoveContainer" containerID="cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d"
Sep 17 16:41:54.035531 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.035509    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d"} err="failed to get container status \"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d\": rpc error: code = NotFound desc = could not find container \"cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d\": container with ID starting with cf3f9f25dc34333cba3c3939fa440e72e5d85cff54cf3f31ba0a44b6385bd07d not found: ID does not exist"
Sep 17 16:41:54.035611 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.035532    2585 scope.go:117] "RemoveContainer" containerID="37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc"
Sep 17 16:41:54.035808 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:54.035788    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc"} err="failed to get container status \"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc\": rpc error: code = NotFound desc = could not find container \"37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc\": container with ID starting with 37d7f58a5ead804e7ea7227dcd72bb62b0caf6b9bc2710ae1b9bc27a631ea9bc not found: ID does not exist"
Sep 17 16:41:55.787758 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:41:55.787723    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" path="/var/lib/kubelet/pods/e0fbd696-8cb1-49c0-a7d4-4899db88130a/volumes"
Sep 17 16:42:53.475364 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475326    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"]
Sep 17 16:42:53.475950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475791    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kube-rbac-proxy"
Sep 17 16:42:53.475950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475811    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kube-rbac-proxy"
Sep 17 16:42:53.475950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475831    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="storage-initializer"
Sep 17 16:42:53.475950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475842    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="storage-initializer"
Sep 17 16:42:53.475950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475855    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kserve-container"
Sep 17 16:42:53.475950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475865    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kserve-container"
Sep 17 16:42:53.475950 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475943    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kserve-container"
Sep 17 16:42:53.476367 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.475955    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0fbd696-8cb1-49c0-a7d4-4899db88130a" containerName="kube-rbac-proxy"
Sep 17 16:42:53.479777 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.479757    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.483203 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.483163    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 16:42:53.483341 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.483189    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:42:53.483417 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.483354    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 16:42:53.483417 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.483379    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:42:53.483842 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.483817    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:42:53.483842 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.483829    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:42:53.484381 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.484365    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:42:53.498362 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.498336    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"]
Sep 17 16:42:53.649684 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.649647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.649684 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.649684    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b9583a-2092-43a8-8961-8418fd43c61d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.649910 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.649702    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt4h\" (UniqueName: \"kubernetes.io/projected/57b9583a-2092-43a8-8961-8418fd43c61d-kube-api-access-gzt4h\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.649910 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.649773    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b9583a-2092-43a8-8961-8418fd43c61d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.649910 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.649792    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.750780 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.750757    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b9583a-2092-43a8-8961-8418fd43c61d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.750946 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.750793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.750946 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.750831    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.750946 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.750847    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b9583a-2092-43a8-8961-8418fd43c61d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.750946 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.750868    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt4h\" (UniqueName: \"kubernetes.io/projected/57b9583a-2092-43a8-8961-8418fd43c61d-kube-api-access-gzt4h\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.751337 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.751309    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b9583a-2092-43a8-8961-8418fd43c61d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.751564 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.751513    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.751564 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.751517    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.753463 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.753432    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b9583a-2092-43a8-8961-8418fd43c61d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.762804 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.762779    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt4h\" (UniqueName: \"kubernetes.io/projected/57b9583a-2092-43a8-8961-8418fd43c61d-kube-api-access-gzt4h\") pod \"isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:53.790655 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:53.790627    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:42:54.072256 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:54.072233    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"]
Sep 17 16:42:54.074134 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:42:54.074106    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b9583a_2092_43a8_8961_8418fd43c61d.slice/crio-f3fce737d2714ea324882e550517051976097288d7109b2197900e17dce23a10 WatchSource:0}: Error finding container f3fce737d2714ea324882e550517051976097288d7109b2197900e17dce23a10: Status 404 returned error can't find the container with id f3fce737d2714ea324882e550517051976097288d7109b2197900e17dce23a10
Sep 17 16:42:54.259224 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:54.259185    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerStarted","Data":"dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9"}
Sep 17 16:42:54.259406 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:54.259252    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerStarted","Data":"f3fce737d2714ea324882e550517051976097288d7109b2197900e17dce23a10"}
Sep 17 16:42:58.278013 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:58.277975    2585 generic.go:356] "Generic (PLEG): container finished" podID="57b9583a-2092-43a8-8961-8418fd43c61d" containerID="dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9" exitCode=0
Sep 17 16:42:58.278419 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:58.278048    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerDied","Data":"dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9"}
Sep 17 16:42:59.288103 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:42:59.288061    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerStarted","Data":"9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977"}
Sep 17 16:43:01.300946 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:01.300904    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerStarted","Data":"8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505"}
Sep 17 16:43:01.301452 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:01.300975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerStarted","Data":"4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657"}
Sep 17 16:43:01.301452 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:01.301181    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:43:01.301452 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:01.301233    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:43:01.326121 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:01.326062    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podStartSLOduration=1.485936793 podStartE2EDuration="8.326045825s" totalImagesPullingTime="2.799636929s" totalInitContainerRuntime="4.040472103s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:01.323334739 +0000 UTC m=+2947.588849731" watchObservedRunningTime="2026-09-17 16:43:01.326045825 +0000 UTC m=+2947.591560826"
Sep 17 16:43:02.305111 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:02.305077    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:43:08.314604 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:08.314527    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:43:28.316037 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:28.315995    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 16:43:33.791500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:33.791456    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 16:43:38.318419 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:43:38.318388    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:44:08.320114 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:08.320082    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:44:13.555928 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:13.555891    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"]
Sep 17 16:44:13.556358 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:13.556190    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container" containerID="cri-o://9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977" gracePeriod=30
Sep 17 16:44:13.556358 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:13.556242    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy" containerID="cri-o://8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505" gracePeriod=30
Sep 17 16:44:13.556482 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:13.556249    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-agent" containerID="cri-o://4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657" gracePeriod=30
Sep 17 16:44:14.631153 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:14.631113    2585 generic.go:356] "Generic (PLEG): container finished" podID="57b9583a-2092-43a8-8961-8418fd43c61d" containerID="8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505" exitCode=2
Sep 17 16:44:14.631153 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:14.631152    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerDied","Data":"8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505"}
Sep 17 16:44:16.642510 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:16.642478    2585 generic.go:356] "Generic (PLEG): container finished" podID="57b9583a-2092-43a8-8961-8418fd43c61d" containerID="9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977" exitCode=0
Sep 17 16:44:16.642891 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:16.642536    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerDied","Data":"9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977"}
Sep 17 16:44:18.309811 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.309763    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 16:44:18.315258 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.315226    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 16:44:18.725181 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.725092    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"]
Sep 17 16:44:18.728842 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.728821    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.731125 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.731092    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 16:44:18.731280 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.731188    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 16:44:18.738480 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.738450    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"]
Sep 17 16:44:18.771625 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.771579    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.771817 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.771633    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.771817 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.771682    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.771817 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.771748    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjm4\" (UniqueName: \"kubernetes.io/projected/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-api-access-hfjm4\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.771817 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.771771    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.873110 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.873066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.873110 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.873110    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.873409 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.873153    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.873409 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.873217    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjm4\" (UniqueName: \"kubernetes.io/projected/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-api-access-hfjm4\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.873409 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.873243    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.873623 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.873603    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.874063 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.874039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.874111 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.874039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.875872 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.875854    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:18.884438 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:18.884411    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjm4\" (UniqueName: \"kubernetes.io/projected/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-api-access-hfjm4\") pod \"isvc-paddle-predictor-6d5c9fd4f6-kg66k\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:19.039700 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:19.039662    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:19.315683 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:19.315605    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"]
Sep 17 16:44:19.318863 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:44:19.318830    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909c8fc6_9c52_4c23_a8fb_5c0d56ae3c16.slice/crio-c5870806670da3f7572b17987e57e2b05363e849392f0671c2f250a09d48fb61 WatchSource:0}: Error finding container c5870806670da3f7572b17987e57e2b05363e849392f0671c2f250a09d48fb61: Status 404 returned error can't find the container with id c5870806670da3f7572b17987e57e2b05363e849392f0671c2f250a09d48fb61
Sep 17 16:44:19.660821 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:19.660726    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" event={"ID":"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16","Type":"ContainerStarted","Data":"b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64"}
Sep 17 16:44:19.660821 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:19.660786    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" event={"ID":"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16","Type":"ContainerStarted","Data":"c5870806670da3f7572b17987e57e2b05363e849392f0671c2f250a09d48fb61"}
Sep 17 16:44:23.309991 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:23.309944    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 16:44:24.685194 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:24.685155    2585 generic.go:356] "Generic (PLEG): container finished" podID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerID="b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64" exitCode=0
Sep 17 16:44:24.685649 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:24.685230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" event={"ID":"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16","Type":"ContainerDied","Data":"b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64"}
Sep 17 16:44:25.998034 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:25.997997    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:44:26.001747 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:26.001724    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:44:28.309497 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:28.309444    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 16:44:28.309996 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:28.309600    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:44:28.315214 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:28.315175    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 16:44:33.309510 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:33.309458    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 16:44:36.747831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:36.747792    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" event={"ID":"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16","Type":"ContainerStarted","Data":"76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4"}
Sep 17 16:44:36.748345 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:36.747875    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" event={"ID":"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16","Type":"ContainerStarted","Data":"81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d"}
Sep 17 16:44:36.748345 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:36.748132    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:36.748492 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:36.748357    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:36.749977 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:36.749941    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:44:37.752629 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:37.752585    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:44:38.309611 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:38.309573    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 16:44:38.315236 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:38.315212    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 16:44:38.315357 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:38.315343    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:44:38.349951 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:38.349905    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podStartSLOduration=3.4552032759999998 podStartE2EDuration="20.349886089s" totalImagesPullingTime="11.737328891s" totalInitContainerRuntime="5.157353922s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:36.780117015 +0000 UTC m=+3043.045632016" watchObservedRunningTime="2026-09-17 16:44:38.349886089 +0000 UTC m=+3044.615401084"
Sep 17 16:44:42.757551 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:42.757524    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:44:42.758021 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:42.757957    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:44:43.309781 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.309736    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 16:44:43.750115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.750086    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:44:43.783078 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.783052    2585 generic.go:356] "Generic (PLEG): container finished" podID="57b9583a-2092-43a8-8961-8418fd43c61d" containerID="4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657" exitCode=0
Sep 17 16:44:43.783473 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.783178    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"
Sep 17 16:44:43.785047 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.785029    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerDied","Data":"4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657"}
Sep 17 16:44:43.785133 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.785086    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5" event={"ID":"57b9583a-2092-43a8-8961-8418fd43c61d","Type":"ContainerDied","Data":"f3fce737d2714ea324882e550517051976097288d7109b2197900e17dce23a10"}
Sep 17 16:44:43.785133 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.785119    2585 scope.go:117] "RemoveContainer" containerID="8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505"
Sep 17 16:44:43.793410 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.793390    2585 scope.go:117] "RemoveContainer" containerID="4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657"
Sep 17 16:44:43.800742 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.800724    2585 scope.go:117] "RemoveContainer" containerID="9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977"
Sep 17 16:44:43.807929 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.807913    2585 scope.go:117] "RemoveContainer" containerID="dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9"
Sep 17 16:44:43.815057 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.815042    2585 scope.go:117] "RemoveContainer" containerID="8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505"
Sep 17 16:44:43.815277 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:44:43.815257    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505\": container with ID starting with 8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505 not found: ID does not exist" containerID="8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505"
Sep 17 16:44:43.815364 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.815302    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505"} err="failed to get container status \"8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505\": rpc error: code = NotFound desc = could not find container \"8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505\": container with ID starting with 8f46bb3d99f49c5b7cbcae5659b34353941888d7204ea9209549dadf255e6505 not found: ID does not exist"
Sep 17 16:44:43.815364 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.815320    2585 scope.go:117] "RemoveContainer" containerID="4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657"
Sep 17 16:44:43.815580 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:44:43.815564    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657\": container with ID starting with 4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657 not found: ID does not exist" containerID="4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657"
Sep 17 16:44:43.815631 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.815589    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657"} err="failed to get container status \"4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657\": rpc error: code = NotFound desc = could not find container \"4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657\": container with ID starting with 4447297d14b387d07ebea0e0c829a0cf0aeea331966459f8c48545428c52d657 not found: ID does not exist"
Sep 17 16:44:43.815631 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.815606    2585 scope.go:117] "RemoveContainer" containerID="9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977"
Sep 17 16:44:43.815843 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:44:43.815824    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977\": container with ID starting with 9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977 not found: ID does not exist" containerID="9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977"
Sep 17 16:44:43.815893 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.815848    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977"} err="failed to get container status \"9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977\": rpc error: code = NotFound desc = could not find container \"9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977\": container with ID starting with 9bb41cdcb2065c3157938d5cf7b34c70b25c32073b7ee735dd02625dab82b977 not found: ID does not exist"
Sep 17 16:44:43.815893 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.815863    2585 scope.go:117] "RemoveContainer" containerID="dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9"
Sep 17 16:44:43.816100 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:44:43.816086    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9\": container with ID starting with dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9 not found: ID does not exist" containerID="dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9"
Sep 17 16:44:43.816141 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.816103    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9"} err="failed to get container status \"dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9\": rpc error: code = NotFound desc = could not find container \"dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9\": container with ID starting with dacd66de1a8b63416c1586027ed29d1529e2d92be1853b9b03b4199354eda3a9 not found: ID does not exist"
Sep 17 16:44:43.906727 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.906698    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b9583a-2092-43a8-8961-8418fd43c61d-kserve-provision-location\") pod \"57b9583a-2092-43a8-8961-8418fd43c61d\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") "
Sep 17 16:44:43.906865 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.906741    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzt4h\" (UniqueName: \"kubernetes.io/projected/57b9583a-2092-43a8-8961-8418fd43c61d-kube-api-access-gzt4h\") pod \"57b9583a-2092-43a8-8961-8418fd43c61d\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") "
Sep 17 16:44:43.906865 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.906837    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b9583a-2092-43a8-8961-8418fd43c61d-proxy-tls\") pod \"57b9583a-2092-43a8-8961-8418fd43c61d\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") "
Sep 17 16:44:43.906966 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.906907    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-kube-rbac-proxy-sar-config\") pod \"57b9583a-2092-43a8-8961-8418fd43c61d\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") "
Sep 17 16:44:43.906966 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.906945    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-cabundle-cert\") pod \"57b9583a-2092-43a8-8961-8418fd43c61d\" (UID: \"57b9583a-2092-43a8-8961-8418fd43c61d\") "
Sep 17 16:44:43.907068 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.907036    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b9583a-2092-43a8-8961-8418fd43c61d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57b9583a-2092-43a8-8961-8418fd43c61d" (UID: "57b9583a-2092-43a8-8961-8418fd43c61d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:43.907217 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.907199    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b9583a-2092-43a8-8961-8418fd43c61d-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:44:43.907335 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.907311    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57b9583a-2092-43a8-8961-8418fd43c61d" (UID: "57b9583a-2092-43a8-8961-8418fd43c61d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:43.907398 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.907368    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57b9583a-2092-43a8-8961-8418fd43c61d" (UID: "57b9583a-2092-43a8-8961-8418fd43c61d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:43.908943 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.908923    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b9583a-2092-43a8-8961-8418fd43c61d-kube-api-access-gzt4h" (OuterVolumeSpecName: "kube-api-access-gzt4h") pod "57b9583a-2092-43a8-8961-8418fd43c61d" (UID: "57b9583a-2092-43a8-8961-8418fd43c61d"). InnerVolumeSpecName "kube-api-access-gzt4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:43.909013 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:43.909000    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b9583a-2092-43a8-8961-8418fd43c61d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57b9583a-2092-43a8-8961-8418fd43c61d" (UID: "57b9583a-2092-43a8-8961-8418fd43c61d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:44.008569 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:44.008528    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57b9583a-2092-43a8-8961-8418fd43c61d-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:44:44.008569 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:44.008566    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:44:44.008569 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:44.008577    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b9583a-2092-43a8-8961-8418fd43c61d-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:44:44.008780 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:44.008585    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzt4h\" (UniqueName: \"kubernetes.io/projected/57b9583a-2092-43a8-8961-8418fd43c61d-kube-api-access-gzt4h\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:44:44.109090 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:44.109059    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"]
Sep 17 16:44:44.113667 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:44.113643    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-86c9d9c8f5-csxv5"]
Sep 17 16:44:45.782556 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:45.782522    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" path="/var/lib/kubelet/pods/57b9583a-2092-43a8-8961-8418fd43c61d/volumes"
Sep 17 16:44:52.758675 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:44:52.758635    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:45:02.758873 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:02.758821    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:45:12.758809 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:12.758770    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:45:22.759057 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:22.759008    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:45:30.087167 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:30.087131    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"]
Sep 17 16:45:30.087673 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:30.087444    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container" containerID="cri-o://81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d" gracePeriod=30
Sep 17 16:45:30.087673 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:30.087478    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kube-rbac-proxy" containerID="cri-o://76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4" gracePeriod=30
Sep 17 16:45:31.003547 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:31.003508    2585 generic.go:356] "Generic (PLEG): container finished" podID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerID="76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4" exitCode=2
Sep 17 16:45:31.003744 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:31.003588    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" event={"ID":"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16","Type":"ContainerDied","Data":"76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4"}
Sep 17 16:45:32.753207 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:32.753168    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 17 16:45:32.758499 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:32.758462    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:45:33.628464 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.628443    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:45:33.753543 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.753460    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfjm4\" (UniqueName: \"kubernetes.io/projected/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-api-access-hfjm4\") pod \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") "
Sep 17 16:45:33.753543 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.753501    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-rbac-proxy-sar-config\") pod \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") "
Sep 17 16:45:33.753543 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.753540    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kserve-provision-location\") pod \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") "
Sep 17 16:45:33.754156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.753609    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-proxy-tls\") pod \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") "
Sep 17 16:45:33.754156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.753637    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-cabundle-cert\") pod \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\" (UID: \"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16\") "
Sep 17 16:45:33.754156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.754079    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" (UID: "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:33.754270 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.754177    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" (UID: "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:33.755896 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.755875    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-api-access-hfjm4" (OuterVolumeSpecName: "kube-api-access-hfjm4") pod "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" (UID: "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16"). InnerVolumeSpecName "kube-api-access-hfjm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:45:33.755982 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.755931    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" (UID: "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:33.763059 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.763038    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" (UID: "909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:45:33.855118 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.855081    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjm4\" (UniqueName: \"kubernetes.io/projected/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-api-access-hfjm4\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:45:33.855118 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.855112    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:45:33.855118 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.855124    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:45:33.855366 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.855137    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:45:33.855366 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:33.855152    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:45:34.018765 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.018678    2585 generic.go:356] "Generic (PLEG): container finished" podID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerID="81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d" exitCode=0
Sep 17 16:45:34.018765 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.018715    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" event={"ID":"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16","Type":"ContainerDied","Data":"81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d"}
Sep 17 16:45:34.018765 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.018739    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"
Sep 17 16:45:34.019003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.018768    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k" event={"ID":"909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16","Type":"ContainerDied","Data":"c5870806670da3f7572b17987e57e2b05363e849392f0671c2f250a09d48fb61"}
Sep 17 16:45:34.019003 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.018794    2585 scope.go:117] "RemoveContainer" containerID="76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4"
Sep 17 16:45:34.027783 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.027760    2585 scope.go:117] "RemoveContainer" containerID="81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d"
Sep 17 16:45:34.036428 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.036406    2585 scope.go:117] "RemoveContainer" containerID="b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64"
Sep 17 16:45:34.036841 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.036816    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"]
Sep 17 16:45:34.041318 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.041275    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-kg66k"]
Sep 17 16:45:34.049862 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.049836    2585 scope.go:117] "RemoveContainer" containerID="76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4"
Sep 17 16:45:34.050239 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:45:34.050218    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4\": container with ID starting with 76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4 not found: ID does not exist" containerID="76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4"
Sep 17 16:45:34.050359 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.050246    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4"} err="failed to get container status \"76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4\": rpc error: code = NotFound desc = could not find container \"76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4\": container with ID starting with 76d561eb2dc90d589906151b1e45f220e6c1a94759b2d2226341778bd43932e4 not found: ID does not exist"
Sep 17 16:45:34.050359 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.050265    2585 scope.go:117] "RemoveContainer" containerID="81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d"
Sep 17 16:45:34.050699 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:45:34.050678    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d\": container with ID starting with 81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d not found: ID does not exist" containerID="81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d"
Sep 17 16:45:34.050788 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.050705    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d"} err="failed to get container status \"81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d\": rpc error: code = NotFound desc = could not find container \"81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d\": container with ID starting with 81249b2c5ef9657d39cd35530bd2088fec49c7fcd2c3811df9f69b191439177d not found: ID does not exist"
Sep 17 16:45:34.050788 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.050722    2585 scope.go:117] "RemoveContainer" containerID="b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64"
Sep 17 16:45:34.051029 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:45:34.051013    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64\": container with ID starting with b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64 not found: ID does not exist" containerID="b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64"
Sep 17 16:45:34.051124 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:34.051033    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64"} err="failed to get container status \"b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64\": rpc error: code = NotFound desc = could not find container \"b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64\": container with ID starting with b8beacdcdb372ae53ff83f17502d5c609189ee0e3b86ac01cd17c56e3d76bd64 not found: ID does not exist"
Sep 17 16:45:35.788966 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:45:35.788922    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" path="/var/lib/kubelet/pods/909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16/volumes"
Sep 17 16:46:26.441723 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.441671    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"]
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442021    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442032    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442042    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442047    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442068    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442075    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442083    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="storage-initializer"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442089    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="storage-initializer"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442102    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kube-rbac-proxy"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442107    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kube-rbac-proxy"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442116    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-agent"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442121    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-agent"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442127    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="storage-initializer"
Sep 17 16:46:26.442173 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442133    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="storage-initializer"
Sep 17 16:46:26.442634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442187    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kube-rbac-proxy"
Sep 17 16:46:26.442634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442194    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kserve-container"
Sep 17 16:46:26.442634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442200    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-agent"
Sep 17 16:46:26.442634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442208    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b9583a-2092-43a8-8961-8418fd43c61d" containerName="kserve-container"
Sep 17 16:46:26.442634 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.442215    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="909c8fc6-9c52-4c23-a8fb-5c0d56ae3c16" containerName="kube-rbac-proxy"
Sep 17 16:46:26.445393 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.445376    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.449263 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.449233    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 16:46:26.449439 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.449246    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 16:46:26.449575 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.449558    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:46:26.449644 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.449593    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:46:26.450328 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.450311    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 16:46:26.450404 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.450313    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:46:26.462137 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.462105    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"]
Sep 17 16:46:26.605010 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.604967    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.605249 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.605023    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.605249 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.605077    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjnl\" (UniqueName: \"kubernetes.io/projected/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-api-access-2tjnl\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.605249 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.605096    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.706292 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.706175    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.706292 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.706242    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.706585 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.706304    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjnl\" (UniqueName: \"kubernetes.io/projected/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-api-access-2tjnl\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.706585 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.706334    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.706585 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:46:26.706500    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 16:46:26.706585 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:46:26.706569    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls podName:2f1cb465-e073-4b03-81d9-8222dd6c3e60 nodeName:}" failed. No retries permitted until 2026-09-17 16:46:27.206547712 +0000 UTC m=+3153.472062693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r" (UID: "2f1cb465-e073-4b03-81d9-8222dd6c3e60") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 16:46:26.706867 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.706730    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.706920 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.706898    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:26.717836 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:26.717788    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjnl\" (UniqueName: \"kubernetes.io/projected/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-api-access-2tjnl\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:27.210378 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:27.210337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:27.212978 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:27.212949    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:27.355529 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:27.355484    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:27.639473 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:27.639408    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"]
Sep 17 16:46:27.641964 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:46:27.641929    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1cb465_e073_4b03_81d9_8222dd6c3e60.slice/crio-480cfcc624196138c94b65f94f160916474b9565a09924fa356e9b3916c0b159 WatchSource:0}: Error finding container 480cfcc624196138c94b65f94f160916474b9565a09924fa356e9b3916c0b159: Status 404 returned error can't find the container with id 480cfcc624196138c94b65f94f160916474b9565a09924fa356e9b3916c0b159
Sep 17 16:46:28.254831 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:28.254794    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r" event={"ID":"2f1cb465-e073-4b03-81d9-8222dd6c3e60","Type":"ContainerStarted","Data":"0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596"}
Sep 17 16:46:28.255023 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:28.254855    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r" event={"ID":"2f1cb465-e073-4b03-81d9-8222dd6c3e60","Type":"ContainerStarted","Data":"480cfcc624196138c94b65f94f160916474b9565a09924fa356e9b3916c0b159"}
Sep 17 16:46:33.278905 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:33.278872    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r_2f1cb465-e073-4b03-81d9-8222dd6c3e60/storage-initializer/0.log"
Sep 17 16:46:33.279304 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:33.278916    2585 generic.go:356] "Generic (PLEG): container finished" podID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerID="0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596" exitCode=1
Sep 17 16:46:33.279304 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:33.278997    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r" event={"ID":"2f1cb465-e073-4b03-81d9-8222dd6c3e60","Type":"ContainerDied","Data":"0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596"}
Sep 17 16:46:34.286177 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:34.286146    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r_2f1cb465-e073-4b03-81d9-8222dd6c3e60/storage-initializer/0.log"
Sep 17 16:46:34.286587 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:34.286260    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r" event={"ID":"2f1cb465-e073-4b03-81d9-8222dd6c3e60","Type":"ContainerStarted","Data":"56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef"}
Sep 17 16:46:36.415769 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:36.415732    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"]
Sep 17 16:46:36.416191 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:36.415942    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r" podUID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerName="storage-initializer" containerID="cri-o://56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef" gracePeriod=30
Sep 17 16:46:38.669217 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.669190    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r_2f1cb465-e073-4b03-81d9-8222dd6c3e60/storage-initializer/1.log"
Sep 17 16:46:38.669685 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.669665    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r_2f1cb465-e073-4b03-81d9-8222dd6c3e60/storage-initializer/0.log"
Sep 17 16:46:38.672096 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.672078    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:38.707149 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.707033    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-rbac-proxy-sar-config\") pod \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") "
Sep 17 16:46:38.707149 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.707096    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls\") pod \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") "
Sep 17 16:46:38.707149 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.707125    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tjnl\" (UniqueName: \"kubernetes.io/projected/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-api-access-2tjnl\") pod \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") "
Sep 17 16:46:38.707472 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.707176    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kserve-provision-location\") pod \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\" (UID: \"2f1cb465-e073-4b03-81d9-8222dd6c3e60\") "
Sep 17 16:46:38.707528 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.707503    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f1cb465-e073-4b03-81d9-8222dd6c3e60" (UID: "2f1cb465-e073-4b03-81d9-8222dd6c3e60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:38.707589 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.707520    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f1cb465-e073-4b03-81d9-8222dd6c3e60" (UID: "2f1cb465-e073-4b03-81d9-8222dd6c3e60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:46:38.709573 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.709544    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f1cb465-e073-4b03-81d9-8222dd6c3e60" (UID: "2f1cb465-e073-4b03-81d9-8222dd6c3e60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:46:38.709573 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.709555    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-api-access-2tjnl" (OuterVolumeSpecName: "kube-api-access-2tjnl") pod "2f1cb465-e073-4b03-81d9-8222dd6c3e60" (UID: "2f1cb465-e073-4b03-81d9-8222dd6c3e60"). InnerVolumeSpecName "kube-api-access-2tjnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:46:38.808377 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.808335    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:46:38.808377 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.808372    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f1cb465-e073-4b03-81d9-8222dd6c3e60-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:46:38.808377 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.808383    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjnl\" (UniqueName: \"kubernetes.io/projected/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kube-api-access-2tjnl\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:46:38.808377 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:38.808391    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f1cb465-e073-4b03-81d9-8222dd6c3e60-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:46:39.310695 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.310669    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r_2f1cb465-e073-4b03-81d9-8222dd6c3e60/storage-initializer/1.log"
Sep 17 16:46:39.311065 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.311052    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r_2f1cb465-e073-4b03-81d9-8222dd6c3e60/storage-initializer/0.log"
Sep 17 16:46:39.311141 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.311089    2585 generic.go:356] "Generic (PLEG): container finished" podID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerID="56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef" exitCode=1
Sep 17 16:46:39.311141 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.311118    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r" event={"ID":"2f1cb465-e073-4b03-81d9-8222dd6c3e60","Type":"ContainerDied","Data":"56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef"}
Sep 17 16:46:39.311266 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.311157    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"
Sep 17 16:46:39.311266 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.311176    2585 scope.go:117] "RemoveContainer" containerID="56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef"
Sep 17 16:46:39.311402 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.311162    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r" event={"ID":"2f1cb465-e073-4b03-81d9-8222dd6c3e60","Type":"ContainerDied","Data":"480cfcc624196138c94b65f94f160916474b9565a09924fa356e9b3916c0b159"}
Sep 17 16:46:39.322184 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.322121    2585 scope.go:117] "RemoveContainer" containerID="0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596"
Sep 17 16:46:39.331168 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.331147    2585 scope.go:117] "RemoveContainer" containerID="56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef"
Sep 17 16:46:39.331501 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:46:39.331483    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef\": container with ID starting with 56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef not found: ID does not exist" containerID="56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef"
Sep 17 16:46:39.331556 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.331510    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef"} err="failed to get container status \"56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef\": rpc error: code = NotFound desc = could not find container \"56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef\": container with ID starting with 56075b01effde85adda90a3df98c9d80c3bdccdfd49317c2b4da783a273fafef not found: ID does not exist"
Sep 17 16:46:39.331556 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.331529    2585 scope.go:117] "RemoveContainer" containerID="0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596"
Sep 17 16:46:39.331759 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:46:39.331746    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596\": container with ID starting with 0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596 not found: ID does not exist" containerID="0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596"
Sep 17 16:46:39.331801 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.331762    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596"} err="failed to get container status \"0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596\": rpc error: code = NotFound desc = could not find container \"0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596\": container with ID starting with 0a8c77d33d0e7cc460f291077234050ba144fcb5999eb65b9a6043288ce27596 not found: ID does not exist"
Sep 17 16:46:39.362097 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.362062    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"]
Sep 17 16:46:39.369145 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.369095    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-fcbfcd549-g254r"]
Sep 17 16:46:39.782407 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:46:39.782371    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" path="/var/lib/kubelet/pods/2f1cb465-e073-4b03-81d9-8222dd6c3e60/volumes"
Sep 17 16:48:08.884444 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.884404    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"]
Sep 17 16:48:08.885069 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.884726    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerName="storage-initializer"
Sep 17 16:48:08.885069 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.884738    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerName="storage-initializer"
Sep 17 16:48:08.885069 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.884806    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerName="storage-initializer"
Sep 17 16:48:08.885069 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.884816    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerName="storage-initializer"
Sep 17 16:48:08.885069 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.884872    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerName="storage-initializer"
Sep 17 16:48:08.885069 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.884877    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1cb465-e073-4b03-81d9-8222dd6c3e60" containerName="storage-initializer"
Sep 17 16:48:08.888085 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.888066    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:08.891138 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.891116    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:48:08.891646 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.891627    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 16:48:08.892235 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.892192    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:48:08.892235 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.892207    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:48:08.892466 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.892255    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:48:08.892466 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.892208    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 16:48:08.892466 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.892213    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:48:08.899551 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.899525    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"]
Sep 17 16:48:08.991927 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.991891    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjmg\" (UniqueName: \"kubernetes.io/projected/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-api-access-nbjmg\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:08.992118 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.991947    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:08.992118 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.992004    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:08.992118 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.992109    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:08.992232 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:08.992151    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.093500 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.093458    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.093705 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.093511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.093705 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.093558    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjmg\" (UniqueName: \"kubernetes.io/projected/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-api-access-nbjmg\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.093705 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.093583    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.093705 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:48:09.093612    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 17 16:48:09.093705 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:48:09.093694    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls podName:602c943a-c2bf-4f51-8f0e-7b6c2d21d874 nodeName:}" failed. No retries permitted until 2026-09-17 16:48:09.59367697 +0000 UTC m=+3255.859191947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls") pod "isvc-pmml-predictor-8674795f59-t6hxz" (UID: "602c943a-c2bf-4f51-8f0e-7b6c2d21d874") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 17 16:48:09.094045 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.093617    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.094045 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.094031    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.094275 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.094256    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.094400 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.094383    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.106754 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.106714    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjmg\" (UniqueName: \"kubernetes.io/projected/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-api-access-nbjmg\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.599875 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.599833    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.602502 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.602474    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-t6hxz\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:09.800072 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:09.800026    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:10.079795 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:10.079761    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"]
Sep 17 16:48:10.082972 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:48:10.082939    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602c943a_c2bf_4f51_8f0e_7b6c2d21d874.slice/crio-89159f5a0dd8a20809713527709c47ca8711c1c840ef6a1aaf63baf12cf9645f WatchSource:0}: Error finding container 89159f5a0dd8a20809713527709c47ca8711c1c840ef6a1aaf63baf12cf9645f: Status 404 returned error can't find the container with id 89159f5a0dd8a20809713527709c47ca8711c1c840ef6a1aaf63baf12cf9645f
Sep 17 16:48:10.704729 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:10.704688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" event={"ID":"602c943a-c2bf-4f51-8f0e-7b6c2d21d874","Type":"ContainerStarted","Data":"8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45"}
Sep 17 16:48:10.704916 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:10.704762    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" event={"ID":"602c943a-c2bf-4f51-8f0e-7b6c2d21d874","Type":"ContainerStarted","Data":"89159f5a0dd8a20809713527709c47ca8711c1c840ef6a1aaf63baf12cf9645f"}
Sep 17 16:48:14.723402 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:14.723309    2585 generic.go:356] "Generic (PLEG): container finished" podID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerID="8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45" exitCode=0
Sep 17 16:48:14.723402 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:14.723325    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" event={"ID":"602c943a-c2bf-4f51-8f0e-7b6c2d21d874","Type":"ContainerDied","Data":"8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45"}
Sep 17 16:48:14.724773 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:14.724755    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:48:22.768421 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:22.768380    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" event={"ID":"602c943a-c2bf-4f51-8f0e-7b6c2d21d874","Type":"ContainerStarted","Data":"edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea"}
Sep 17 16:48:22.768819 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:22.768488    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" event={"ID":"602c943a-c2bf-4f51-8f0e-7b6c2d21d874","Type":"ContainerStarted","Data":"3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48"}
Sep 17 16:48:22.768819 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:22.768789    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:22.768951 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:22.768930    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:22.770158 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:22.770135    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:48:22.794145 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:22.794086    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podStartSLOduration=3.178978763 podStartE2EDuration="14.794070068s" totalImagesPullingTime="7.40028417s" totalInitContainerRuntime="4.214807135s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:22.791972335 +0000 UTC m=+3269.057487336" watchObservedRunningTime="2026-09-17 16:48:22.794070068 +0000 UTC m=+3269.059585067"
Sep 17 16:48:23.773606 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:23.773566    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:48:28.778534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:28.778503    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:48:28.778968 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:28.778945    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:48:38.779507 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:38.779466    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:48:48.779069 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:48.779029    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:48:58.779355 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:48:58.779312    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:49:08.779858 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:49:08.779768    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:49:18.779806 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:49:18.779767    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:49:26.030301 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:49:26.030268    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:49:26.036122 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:49:26.036091    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:49:28.779858 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:49:28.779815    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:49:38.779738 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:49:38.779698    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:49:46.778085 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:49:46.778039    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 16:49:56.779494 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:49:56.779461    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:50:00.666643 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:00.666609    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"]
Sep 17 16:50:00.667107 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:00.666887    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container" containerID="cri-o://3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48" gracePeriod=30
Sep 17 16:50:00.667107 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:00.666954    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kube-rbac-proxy" containerID="cri-o://edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea" gracePeriod=30
Sep 17 16:50:01.198936 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:01.198904    2585 generic.go:356] "Generic (PLEG): container finished" podID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerID="edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea" exitCode=2
Sep 17 16:50:01.199114 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:01.198976    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" event={"ID":"602c943a-c2bf-4f51-8f0e-7b6c2d21d874","Type":"ContainerDied","Data":"edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea"}
Sep 17 16:50:03.774428 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:03.774386    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 16:50:05.113522 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.113491    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:50:05.196087 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.195996    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls\") pod \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") "
Sep 17 16:50:05.196087 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.196036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kserve-provision-location\") pod \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") "
Sep 17 16:50:05.196087 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.196060    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-rbac-proxy-sar-config\") pod \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") "
Sep 17 16:50:05.196412 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.196182    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-cabundle-cert\") pod \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") "
Sep 17 16:50:05.196412 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.196254    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbjmg\" (UniqueName: \"kubernetes.io/projected/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-api-access-nbjmg\") pod \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\" (UID: \"602c943a-c2bf-4f51-8f0e-7b6c2d21d874\") "
Sep 17 16:50:05.196412 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.196390    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "602c943a-c2bf-4f51-8f0e-7b6c2d21d874" (UID: "602c943a-c2bf-4f51-8f0e-7b6c2d21d874"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:50:05.196592 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.196527    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "602c943a-c2bf-4f51-8f0e-7b6c2d21d874" (UID: "602c943a-c2bf-4f51-8f0e-7b6c2d21d874"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:50:05.196592 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.196584    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kserve-provision-location\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:50:05.196687 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.196611    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "602c943a-c2bf-4f51-8f0e-7b6c2d21d874" (UID: "602c943a-c2bf-4f51-8f0e-7b6c2d21d874"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:50:05.198493 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.198471    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-api-access-nbjmg" (OuterVolumeSpecName: "kube-api-access-nbjmg") pod "602c943a-c2bf-4f51-8f0e-7b6c2d21d874" (UID: "602c943a-c2bf-4f51-8f0e-7b6c2d21d874"). InnerVolumeSpecName "kube-api-access-nbjmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:50:05.198611 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.198590    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "602c943a-c2bf-4f51-8f0e-7b6c2d21d874" (UID: "602c943a-c2bf-4f51-8f0e-7b6c2d21d874"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:50:05.218735 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.218700    2585 generic.go:356] "Generic (PLEG): container finished" podID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerID="3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48" exitCode=0
Sep 17 16:50:05.218908 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.218768    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"
Sep 17 16:50:05.218908 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.218772    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" event={"ID":"602c943a-c2bf-4f51-8f0e-7b6c2d21d874","Type":"ContainerDied","Data":"3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48"}
Sep 17 16:50:05.218908 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.218835    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz" event={"ID":"602c943a-c2bf-4f51-8f0e-7b6c2d21d874","Type":"ContainerDied","Data":"89159f5a0dd8a20809713527709c47ca8711c1c840ef6a1aaf63baf12cf9645f"}
Sep 17 16:50:05.218908 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.218874    2585 scope.go:117] "RemoveContainer" containerID="edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea"
Sep 17 16:50:05.228208 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.228185    2585 scope.go:117] "RemoveContainer" containerID="3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48"
Sep 17 16:50:05.238622 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.238597    2585 scope.go:117] "RemoveContainer" containerID="8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45"
Sep 17 16:50:05.247776 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.247745    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"]
Sep 17 16:50:05.248823 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.248799    2585 scope.go:117] "RemoveContainer" containerID="edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea"
Sep 17 16:50:05.249196 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:50:05.249177    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea\": container with ID starting with edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea not found: ID does not exist" containerID="edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea"
Sep 17 16:50:05.249271 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.249205    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea"} err="failed to get container status \"edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea\": rpc error: code = NotFound desc = could not find container \"edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea\": container with ID starting with edbd03d4255a6edfbcee71ca3adba9f177cd2b18939406e7919c49c76fbb27ea not found: ID does not exist"
Sep 17 16:50:05.249271 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.249226    2585 scope.go:117] "RemoveContainer" containerID="3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48"
Sep 17 16:50:05.249521 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:50:05.249500    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48\": container with ID starting with 3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48 not found: ID does not exist" containerID="3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48"
Sep 17 16:50:05.249589 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.249523    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-t6hxz"]
Sep 17 16:50:05.249589 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.249526    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48"} err="failed to get container status \"3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48\": rpc error: code = NotFound desc = could not find container \"3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48\": container with ID starting with 3ab83eb6da3229d9dec52cef427a27c4160391f3d5d685d004e3e780e82cce48 not found: ID does not exist"
Sep 17 16:50:05.249589 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.249581    2585 scope.go:117] "RemoveContainer" containerID="8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45"
Sep 17 16:50:05.249873 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:50:05.249855    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45\": container with ID starting with 8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45 not found: ID does not exist" containerID="8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45"
Sep 17 16:50:05.249916 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.249876    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45"} err="failed to get container status \"8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45\": rpc error: code = NotFound desc = could not find container \"8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45\": container with ID starting with 8eadd0aeb5c3fa8a0f338ea46dc8b00e2c9c5abb6a53cd204b72e7e3ca5a5c45 not found: ID does not exist"
Sep 17 16:50:05.296993 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.296954    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-proxy-tls\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:50:05.296993 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.296983    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:50:05.296993 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.296993    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-cabundle-cert\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:50:05.296993 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.297005    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbjmg\" (UniqueName: \"kubernetes.io/projected/602c943a-c2bf-4f51-8f0e-7b6c2d21d874-kube-api-access-nbjmg\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:50:05.782813 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:50:05.782776    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" path="/var/lib/kubelet/pods/602c943a-c2bf-4f51-8f0e-7b6c2d21d874/volumes"
Sep 17 16:52:30.933448 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933412    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6xm7/must-gather-mq76f"]
Sep 17 16:52:30.933890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933717    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container"
Sep 17 16:52:30.933890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933728    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container"
Sep 17 16:52:30.933890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933750    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="storage-initializer"
Sep 17 16:52:30.933890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933757    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="storage-initializer"
Sep 17 16:52:30.933890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933768    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kube-rbac-proxy"
Sep 17 16:52:30.933890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933773    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kube-rbac-proxy"
Sep 17 16:52:30.933890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933825    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kserve-container"
Sep 17 16:52:30.933890 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.933835    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="602c943a-c2bf-4f51-8f0e-7b6c2d21d874" containerName="kube-rbac-proxy"
Sep 17 16:52:30.936830 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.936814    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:52:30.943399 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.943070    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6xm7\"/\"kube-root-ca.crt\""
Sep 17 16:52:30.943399 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.943161    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6xm7\"/\"openshift-service-ca.crt\""
Sep 17 16:52:30.949897 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.949874    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6xm7/must-gather-mq76f"]
Sep 17 16:52:30.982108 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.982073    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6757e90c-e3e9-46b5-8352-615ced7097ea-must-gather-output\") pod \"must-gather-mq76f\" (UID: \"6757e90c-e3e9-46b5-8352-615ced7097ea\") " pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:52:30.982108 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:30.982108    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqzml\" (UniqueName: \"kubernetes.io/projected/6757e90c-e3e9-46b5-8352-615ced7097ea-kube-api-access-dqzml\") pod \"must-gather-mq76f\" (UID: \"6757e90c-e3e9-46b5-8352-615ced7097ea\") " pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:52:31.083534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:31.083494    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6757e90c-e3e9-46b5-8352-615ced7097ea-must-gather-output\") pod \"must-gather-mq76f\" (UID: \"6757e90c-e3e9-46b5-8352-615ced7097ea\") " pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:52:31.083534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:31.083537    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqzml\" (UniqueName: \"kubernetes.io/projected/6757e90c-e3e9-46b5-8352-615ced7097ea-kube-api-access-dqzml\") pod \"must-gather-mq76f\" (UID: \"6757e90c-e3e9-46b5-8352-615ced7097ea\") " pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:52:31.083843 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:31.083824    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6757e90c-e3e9-46b5-8352-615ced7097ea-must-gather-output\") pod \"must-gather-mq76f\" (UID: \"6757e90c-e3e9-46b5-8352-615ced7097ea\") " pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:52:31.105570 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:31.105545    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqzml\" (UniqueName: \"kubernetes.io/projected/6757e90c-e3e9-46b5-8352-615ced7097ea-kube-api-access-dqzml\") pod \"must-gather-mq76f\" (UID: \"6757e90c-e3e9-46b5-8352-615ced7097ea\") " pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:52:31.245653 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:31.245564    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:52:31.521057 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:31.521027    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6xm7/must-gather-mq76f"]
Sep 17 16:52:31.523033 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:52:31.523006    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6757e90c_e3e9_46b5_8352_615ced7097ea.slice/crio-feedb8e7d58a57a713e0aa09c422f31bdbaddc483644ea476cc04d964bffce24 WatchSource:0}: Error finding container feedb8e7d58a57a713e0aa09c422f31bdbaddc483644ea476cc04d964bffce24: Status 404 returned error can't find the container with id feedb8e7d58a57a713e0aa09c422f31bdbaddc483644ea476cc04d964bffce24
Sep 17 16:52:31.848955 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:31.848856    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6xm7/must-gather-mq76f" event={"ID":"6757e90c-e3e9-46b5-8352-615ced7097ea","Type":"ContainerStarted","Data":"feedb8e7d58a57a713e0aa09c422f31bdbaddc483644ea476cc04d964bffce24"}
Sep 17 16:52:38.887922 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:38.887883    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6xm7/must-gather-mq76f" event={"ID":"6757e90c-e3e9-46b5-8352-615ced7097ea","Type":"ContainerStarted","Data":"a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227"}
Sep 17 16:52:38.887922 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:38.887925    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6xm7/must-gather-mq76f" event={"ID":"6757e90c-e3e9-46b5-8352-615ced7097ea","Type":"ContainerStarted","Data":"c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177"}
Sep 17 16:52:38.925154 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:52:38.925102    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6xm7/must-gather-mq76f" podStartSLOduration=2.690466688 podStartE2EDuration="8.925085666s" totalImagesPullingTime="6.234618978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:38.92404877 +0000 UTC m=+3525.189563771" watchObservedRunningTime="2026-09-17 16:52:38.925085666 +0000 UTC m=+3525.190600718"
Sep 17 16:53:08.027351 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:08.027318    2585 generic.go:356] "Generic (PLEG): container finished" podID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerID="a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227" exitCode=0
Sep 17 16:53:08.027756 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:08.027382    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6xm7/must-gather-mq76f" event={"ID":"6757e90c-e3e9-46b5-8352-615ced7097ea","Type":"ContainerDied","Data":"a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227"}
Sep 17 16:53:08.027756 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:08.027681    2585 scope.go:117] "RemoveContainer" containerID="a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227"
Sep 17 16:53:08.859279 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:08.859255    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m6xm7_must-gather-mq76f_6757e90c-e3e9-46b5-8352-615ced7097ea/gather/0.log"
Sep 17 16:53:09.656628 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.656588    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-94gf4/must-gather-2bdl4"]
Sep 17 16:53:09.660347 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.660327    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-94gf4/must-gather-2bdl4"
Sep 17 16:53:09.663461 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.663432    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-94gf4\"/\"kube-root-ca.crt\""
Sep 17 16:53:09.663555 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.663478    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-94gf4\"/\"default-dockercfg-fjpj5\""
Sep 17 16:53:09.663604 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.663554    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-94gf4\"/\"openshift-service-ca.crt\""
Sep 17 16:53:09.673919 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.673896    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-94gf4/must-gather-2bdl4"]
Sep 17 16:53:09.734690 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.734662    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmr4\" (UniqueName: \"kubernetes.io/projected/815e0c7e-d3db-4948-a60e-54ffd36c1dcb-kube-api-access-6kmr4\") pod \"must-gather-2bdl4\" (UID: \"815e0c7e-d3db-4948-a60e-54ffd36c1dcb\") " pod="openshift-must-gather-94gf4/must-gather-2bdl4"
Sep 17 16:53:09.734853 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.734739    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/815e0c7e-d3db-4948-a60e-54ffd36c1dcb-must-gather-output\") pod \"must-gather-2bdl4\" (UID: \"815e0c7e-d3db-4948-a60e-54ffd36c1dcb\") " pod="openshift-must-gather-94gf4/must-gather-2bdl4"
Sep 17 16:53:09.836036 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.836002    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/815e0c7e-d3db-4948-a60e-54ffd36c1dcb-must-gather-output\") pod \"must-gather-2bdl4\" (UID: \"815e0c7e-d3db-4948-a60e-54ffd36c1dcb\") " pod="openshift-must-gather-94gf4/must-gather-2bdl4"
Sep 17 16:53:09.836201 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.836046    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmr4\" (UniqueName: \"kubernetes.io/projected/815e0c7e-d3db-4948-a60e-54ffd36c1dcb-kube-api-access-6kmr4\") pod \"must-gather-2bdl4\" (UID: \"815e0c7e-d3db-4948-a60e-54ffd36c1dcb\") " pod="openshift-must-gather-94gf4/must-gather-2bdl4"
Sep 17 16:53:09.836403 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.836381    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/815e0c7e-d3db-4948-a60e-54ffd36c1dcb-must-gather-output\") pod \"must-gather-2bdl4\" (UID: \"815e0c7e-d3db-4948-a60e-54ffd36c1dcb\") " pod="openshift-must-gather-94gf4/must-gather-2bdl4"
Sep 17 16:53:09.845956 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.845930    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmr4\" (UniqueName: \"kubernetes.io/projected/815e0c7e-d3db-4948-a60e-54ffd36c1dcb-kube-api-access-6kmr4\") pod \"must-gather-2bdl4\" (UID: \"815e0c7e-d3db-4948-a60e-54ffd36c1dcb\") " pod="openshift-must-gather-94gf4/must-gather-2bdl4"
Sep 17 16:53:09.969519 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:09.969429    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-94gf4/must-gather-2bdl4"
Sep 17 16:53:10.242329 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:10.242239    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-94gf4/must-gather-2bdl4"]
Sep 17 16:53:10.244848 ip-10-0-142-18 kubenswrapper[2585]: W0917 16:53:10.244819    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod815e0c7e_d3db_4948_a60e_54ffd36c1dcb.slice/crio-3616c00cd04a483bef82f50ee6e391b31dd4aa055a3ea2eff57e1e072e74d965 WatchSource:0}: Error finding container 3616c00cd04a483bef82f50ee6e391b31dd4aa055a3ea2eff57e1e072e74d965: Status 404 returned error can't find the container with id 3616c00cd04a483bef82f50ee6e391b31dd4aa055a3ea2eff57e1e072e74d965
Sep 17 16:53:11.044012 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:11.043930    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-94gf4/must-gather-2bdl4" event={"ID":"815e0c7e-d3db-4948-a60e-54ffd36c1dcb","Type":"ContainerStarted","Data":"cbeda78b97a5d4ff8148711744c3c68e1935982f7a8ba2cb6cac97bd92ae0f47"}
Sep 17 16:53:11.044012 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:11.043979    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-94gf4/must-gather-2bdl4" event={"ID":"815e0c7e-d3db-4948-a60e-54ffd36c1dcb","Type":"ContainerStarted","Data":"3616c00cd04a483bef82f50ee6e391b31dd4aa055a3ea2eff57e1e072e74d965"}
Sep 17 16:53:12.051663 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:12.051619    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-94gf4/must-gather-2bdl4" event={"ID":"815e0c7e-d3db-4948-a60e-54ffd36c1dcb","Type":"ContainerStarted","Data":"84313cac9f66764500a1a8a2dc73334c3e551bf0350f6aab571d9f9da7b1da6c"}
Sep 17 16:53:12.069000 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:12.068946    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-94gf4/must-gather-2bdl4" podStartSLOduration=2.42022181 podStartE2EDuration="3.068930572s" totalImagesPullingTime="648.708762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:12.06818482 +0000 UTC m=+3558.333699819" watchObservedRunningTime="2026-09-17 16:53:12.068930572 +0000 UTC m=+3558.334445572"
Sep 17 16:53:12.465045 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:12.464971    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q4f6r_0f843638-5eea-4ff9-99e4-33229d7815a3/global-pull-secret-syncer/0.log"
Sep 17 16:53:12.622108 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:12.622070    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s6qlb_572c0334-33e6-444a-80e3-caa407bd897d/konnectivity-agent/0.log"
Sep 17 16:53:12.708807 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:12.708771    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-18.ec2.internal_3bca85661d4e2826a71d7270087fe9ca/haproxy/0.log"
Sep 17 16:53:14.482440 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.482401    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-m6xm7/must-gather-mq76f"]
Sep 17 16:53:14.482922 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.482650    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-m6xm7/must-gather-mq76f" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerName="copy" containerID="cri-o://c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177" gracePeriod=2
Sep 17 16:53:14.496436 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:53:14.495515    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mq76f\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m6xm7\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:53:14.515682 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.508810    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-m6xm7/must-gather-mq76f"]
Sep 17 16:53:14.906192 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.905891    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m6xm7_must-gather-mq76f_6757e90c-e3e9-46b5-8352-615ced7097ea/copy/0.log"
Sep 17 16:53:14.913560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.913310    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:53:14.917506 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:53:14.917450    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mq76f\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m6xm7\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:53:14.992197 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.990524    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqzml\" (UniqueName: \"kubernetes.io/projected/6757e90c-e3e9-46b5-8352-615ced7097ea-kube-api-access-dqzml\") pod \"6757e90c-e3e9-46b5-8352-615ced7097ea\" (UID: \"6757e90c-e3e9-46b5-8352-615ced7097ea\") "
Sep 17 16:53:14.992197 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.990617    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6757e90c-e3e9-46b5-8352-615ced7097ea-must-gather-output\") pod \"6757e90c-e3e9-46b5-8352-615ced7097ea\" (UID: \"6757e90c-e3e9-46b5-8352-615ced7097ea\") "
Sep 17 16:53:14.992197 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.992150    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6757e90c-e3e9-46b5-8352-615ced7097ea-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "6757e90c-e3e9-46b5-8352-615ced7097ea" (UID: "6757e90c-e3e9-46b5-8352-615ced7097ea"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:53:14.994633 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:14.994599    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6757e90c-e3e9-46b5-8352-615ced7097ea-kube-api-access-dqzml" (OuterVolumeSpecName: "kube-api-access-dqzml") pod "6757e90c-e3e9-46b5-8352-615ced7097ea" (UID: "6757e90c-e3e9-46b5-8352-615ced7097ea"). InnerVolumeSpecName "kube-api-access-dqzml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:53:15.074464 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.074417    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-m6xm7_must-gather-mq76f_6757e90c-e3e9-46b5-8352-615ced7097ea/copy/0.log"
Sep 17 16:53:15.074981 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.074945    2585 generic.go:356] "Generic (PLEG): container finished" podID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerID="c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177" exitCode=143
Sep 17 16:53:15.075104 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.075090    2585 scope.go:117] "RemoveContainer" containerID="c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177"
Sep 17 16:53:15.075346 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.075330    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:53:15.080377 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:53:15.080332    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mq76f\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m6xm7\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:53:15.091809 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.091781    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqzml\" (UniqueName: \"kubernetes.io/projected/6757e90c-e3e9-46b5-8352-615ced7097ea-kube-api-access-dqzml\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:53:15.091978 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.091961    2585 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6757e90c-e3e9-46b5-8352-615ced7097ea-must-gather-output\") on node \"ip-10-0-142-18.ec2.internal\" DevicePath \"\""
Sep 17 16:53:15.092435 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.092415    2585 scope.go:117] "RemoveContainer" containerID="a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227"
Sep 17 16:53:15.097035 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:53:15.096993    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mq76f\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m6xm7\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:53:15.147245 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.146906    2585 scope.go:117] "RemoveContainer" containerID="c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177"
Sep 17 16:53:15.147758 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:53:15.147569    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177\": container with ID starting with c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177 not found: ID does not exist" containerID="c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177"
Sep 17 16:53:15.147758 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.147619    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177"} err="failed to get container status \"c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177\": rpc error: code = NotFound desc = could not find container \"c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177\": container with ID starting with c5cda1bea6aa17a3c11e9d10a50e5956fd7f6f264cbaa757b266427b75557177 not found: ID does not exist"
Sep 17 16:53:15.147758 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.147653    2585 scope.go:117] "RemoveContainer" containerID="a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227"
Sep 17 16:53:15.148383 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:53:15.148305    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227\": container with ID starting with a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227 not found: ID does not exist" containerID="a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227"
Sep 17 16:53:15.148383 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.148341    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227"} err="failed to get container status \"a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227\": rpc error: code = NotFound desc = could not find container \"a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227\": container with ID starting with a5d50ee6469f510e3ed0060fd8ab43cf496b729ac5748cf1d92ba27b4db6d227 not found: ID does not exist"
Sep 17 16:53:15.785954 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:15.785870    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" path="/var/lib/kubelet/pods/6757e90c-e3e9-46b5-8352-615ced7097ea/volumes"
Sep 17 16:53:15.789206 ip-10-0-142-18 kubenswrapper[2585]: E0917 16:53:15.789169    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mq76f\" is forbidden: User \"system:node:ip-10-0-142-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-m6xm7\": no relationship found between node 'ip-10-0-142-18.ec2.internal' and this object" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" pod="openshift-must-gather-m6xm7/must-gather-mq76f"
Sep 17 16:53:16.871560 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:16.871527    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778aa706-2049-4d00-b985-8aff84521b90/alertmanager/0.log"
Sep 17 16:53:16.938643 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:16.938557    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778aa706-2049-4d00-b985-8aff84521b90/config-reloader/0.log"
Sep 17 16:53:16.993276 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:16.993253    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778aa706-2049-4d00-b985-8aff84521b90/kube-rbac-proxy-web/0.log"
Sep 17 16:53:17.117637 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:17.117597    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778aa706-2049-4d00-b985-8aff84521b90/kube-rbac-proxy/0.log"
Sep 17 16:53:17.207002 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:17.206927    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778aa706-2049-4d00-b985-8aff84521b90/kube-rbac-proxy-metric/0.log"
Sep 17 16:53:17.386698 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:17.386671    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778aa706-2049-4d00-b985-8aff84521b90/prom-label-proxy/0.log"
Sep 17 16:53:17.440358 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:17.440328    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778aa706-2049-4d00-b985-8aff84521b90/init-config-reloader/0.log"
Sep 17 16:53:17.805156 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:17.805128    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-59f5f9546d-7m2xx_af656f33-b308-4cc7-bd5f-a689eb612866/metrics-server/0.log"
Sep 17 16:53:17.919077 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:17.919050    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-lc9x7_ea5f39cc-4284-4ac5-a72a-75ebdc2d6056/monitoring-plugin/0.log"
Sep 17 16:53:18.469045 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:18.468986    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xdqnb_46b75e77-1806-45e8-bfd9-2708776f5f99/node-exporter/0.log"
Sep 17 16:53:18.559325 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:18.558743    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xdqnb_46b75e77-1806-45e8-bfd9-2708776f5f99/kube-rbac-proxy/0.log"
Sep 17 16:53:18.699246 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:18.699176    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xdqnb_46b75e77-1806-45e8-bfd9-2708776f5f99/init-textfile/0.log"
Sep 17 16:53:18.851503 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:18.851466    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6xbpt_e579e186-6e4e-4617-b942-e7822402a738/kube-rbac-proxy-main/0.log"
Sep 17 16:53:18.893854 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:18.893784    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6xbpt_e579e186-6e4e-4617-b942-e7822402a738/kube-rbac-proxy-self/0.log"
Sep 17 16:53:18.932115 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:18.932052    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-6xbpt_e579e186-6e4e-4617-b942-e7822402a738/openshift-state-metrics/0.log"
Sep 17 16:53:18.987891 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:18.987818    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_492b25cd-faaf-489a-b68f-58369dca35a2/prometheus/0.log"
Sep 17 16:53:19.013073 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.013043    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_492b25cd-faaf-489a-b68f-58369dca35a2/config-reloader/0.log"
Sep 17 16:53:19.044269 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.044234    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_492b25cd-faaf-489a-b68f-58369dca35a2/thanos-sidecar/0.log"
Sep 17 16:53:19.080622 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.080590    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_492b25cd-faaf-489a-b68f-58369dca35a2/kube-rbac-proxy-web/0.log"
Sep 17 16:53:19.115982 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.115950    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_492b25cd-faaf-489a-b68f-58369dca35a2/kube-rbac-proxy/0.log"
Sep 17 16:53:19.170035 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.170001    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_492b25cd-faaf-489a-b68f-58369dca35a2/kube-rbac-proxy-thanos/0.log"
Sep 17 16:53:19.242487 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.242381    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_492b25cd-faaf-489a-b68f-58369dca35a2/init-config-reloader/0.log"
Sep 17 16:53:19.362138 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.362108    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-8h4vk_e3e53b0d-007e-44a2-af3a-35260192466d/prometheus-operator-admission-webhook/0.log"
Sep 17 16:53:19.401396 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.401365    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cbbb68446-qgr42_e2d4036b-7351-48c3-9b8b-2e2cbf05e995/telemeter-client/0.log"
Sep 17 16:53:19.434672 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.434641    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cbbb68446-qgr42_e2d4036b-7351-48c3-9b8b-2e2cbf05e995/reload/0.log"
Sep 17 16:53:19.487050 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:19.487021    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cbbb68446-qgr42_e2d4036b-7351-48c3-9b8b-2e2cbf05e995/kube-rbac-proxy/0.log"
Sep 17 16:53:20.582828 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.582797    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"]
Sep 17 16:53:20.583313 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.583139    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerName="gather"
Sep 17 16:53:20.583313 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.583153    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerName="gather"
Sep 17 16:53:20.583313 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.583163    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerName="copy"
Sep 17 16:53:20.583313 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.583170    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerName="copy"
Sep 17 16:53:20.583313 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.583253    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerName="copy"
Sep 17 16:53:20.583313 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.583263    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6757e90c-e3e9-46b5-8352-615ced7097ea" containerName="gather"
Sep 17 16:53:20.586536 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.586508    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.603723 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.603684    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"]
Sep 17 16:53:20.653984 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.653935    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-lib-modules\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.654189 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.653999    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrqq\" (UniqueName: \"kubernetes.io/projected/b10aa74d-e38f-48bd-b3ed-c610b4617751-kube-api-access-gcrqq\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.654189 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.654071    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-proc\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.654189 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.654102    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-podres\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.654189 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.654170    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-sys\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.743851 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.743812    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-dpcws_498bb02c-2602-4a5b-807d-bf5e8ad38c82/networking-console-plugin/0.log"
Sep 17 16:53:20.755112 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755082    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-lib-modules\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.755264 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755122    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrqq\" (UniqueName: \"kubernetes.io/projected/b10aa74d-e38f-48bd-b3ed-c610b4617751-kube-api-access-gcrqq\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.755264 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755150    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-proc\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.755264 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755259    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-podres\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.755431 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755268    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-lib-modules\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.755431 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755325    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-proc\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.755431 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755353    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-sys\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.755431 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755401    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-podres\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.755601 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.755436    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b10aa74d-e38f-48bd-b3ed-c610b4617751-sys\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.767345 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.767315    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrqq\" (UniqueName: \"kubernetes.io/projected/b10aa74d-e38f-48bd-b3ed-c610b4617751-kube-api-access-gcrqq\") pod \"perf-node-gather-daemonset-jxdbn\" (UID: \"b10aa74d-e38f-48bd-b3ed-c610b4617751\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:20.900835 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:20.900743    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:21.206608 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:21.206122    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"]
Sep 17 16:53:21.270738 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:21.270710    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/2.log"
Sep 17 16:53:21.276306 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:21.276268    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l7rxh_53c5829f-704f-4409-a7f6-95ac75a37a45/console-operator/3.log"
Sep 17 16:53:22.118370 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:22.118325    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn" event={"ID":"b10aa74d-e38f-48bd-b3ed-c610b4617751","Type":"ContainerStarted","Data":"9f761e5a716c4f2c2f6dfbee89aa951aeab13cd522211ab90c151516f86d1947"}
Sep 17 16:53:22.118791 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:22.118398    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn" event={"ID":"b10aa74d-e38f-48bd-b3ed-c610b4617751","Type":"ContainerStarted","Data":"a7f3d70e5b3292d33a18f1800c96dacf8cb691397bbac0fc2cadbf8a2e0268e8"}
Sep 17 16:53:22.118791 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:22.118460    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:22.156987 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:22.156920    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn" podStartSLOduration=2.156901281 podStartE2EDuration="2.156901281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:22.152794616 +0000 UTC m=+3568.418309616" watchObservedRunningTime="2026-09-17 16:53:22.156901281 +0000 UTC m=+3568.422416283"
Sep 17 16:53:23.395791 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:23.395755    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbb7d_49b31cc2-74bc-48f2-8420-9bba8de27ab5/dns/0.log"
Sep 17 16:53:23.420655 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:23.420627    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbb7d_49b31cc2-74bc-48f2-8420-9bba8de27ab5/kube-rbac-proxy/0.log"
Sep 17 16:53:23.502261 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:23.502233    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ksfvw_64748e47-f5f7-47d7-a2cb-0a5ff8ffd3fb/dns-node-resolver/0.log"
Sep 17 16:53:24.092219 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:24.092188    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-prh25_74438d26-e6ab-4d96-a541-0f428f8951f2/node-ca/0.log"
Sep 17 16:53:25.405054 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:25.405015    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v5ms8_48fc9971-ac9e-4939-853c-cb4851a1f652/serve-healthcheck-canary/0.log"
Sep 17 16:53:25.938729 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:25.938698    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5mrxm_3be4d3a6-64d1-4da1-a1e9-59fef88b7345/kube-rbac-proxy/0.log"
Sep 17 16:53:25.970742 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:25.970709    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5mrxm_3be4d3a6-64d1-4da1-a1e9-59fef88b7345/exporter/0.log"
Sep 17 16:53:26.007770 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:26.007741    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5mrxm_3be4d3a6-64d1-4da1-a1e9-59fef88b7345/extractor/0.log"
Sep 17 16:53:28.136967 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:28.136934    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-jxdbn"
Sep 17 16:53:29.265100 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:29.265061    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x6jp6_c77c66a1-2713-4ffc-8f9a-3ec101ba4315/s3-init/0.log"
Sep 17 16:53:29.310397 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:29.310364    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x6jp6_c77c66a1-2713-4ffc-8f9a-3ec101ba4315/download-hf-model/0.log"
Sep 17 16:53:29.397278 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:29.397244    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-cvxvk_8805fe9f-c9c5-447a-bad2-fabad597ad81/s3-tls-init-custom/0.log"
Sep 17 16:53:29.434355 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:29.434247    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-cvxvk_8805fe9f-c9c5-447a-bad2-fabad597ad81/download-hf-model/0.log"
Sep 17 16:53:29.502452 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:29.502424    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-h9vjj_6f75400d-a054-447b-a0ed-b38e302afe1d/s3-tls-init-serving/0.log"
Sep 17 16:53:29.594372 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:29.594346    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-h9vjj_6f75400d-a054-447b-a0ed-b38e302afe1d/download-hf-model/0.log"
Sep 17 16:53:35.426713 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:35.426677    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ts59f_452444d7-ed37-4ddb-b83a-192ee0ef5727/kube-storage-version-migrator-operator/1.log"
Sep 17 16:53:35.427243 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:35.427225    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ts59f_452444d7-ed37-4ddb-b83a-192ee0ef5727/kube-storage-version-migrator-operator/0.log"
Sep 17 16:53:37.076885 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.076806    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s28wz_d6c09c6b-33dd-4de9-aab8-222a50919654/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:53:37.105521 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.105493    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s28wz_d6c09c6b-33dd-4de9-aab8-222a50919654/egress-router-binary-copy/0.log"
Sep 17 16:53:37.128944 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.128920    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s28wz_d6c09c6b-33dd-4de9-aab8-222a50919654/cni-plugins/0.log"
Sep 17 16:53:37.154970 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.154939    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s28wz_d6c09c6b-33dd-4de9-aab8-222a50919654/bond-cni-plugin/0.log"
Sep 17 16:53:37.181534 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.181508    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s28wz_d6c09c6b-33dd-4de9-aab8-222a50919654/routeoverride-cni/0.log"
Sep 17 16:53:37.205989 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.205959    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s28wz_d6c09c6b-33dd-4de9-aab8-222a50919654/whereabouts-cni-bincopy/0.log"
Sep 17 16:53:37.230702 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.230674    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s28wz_d6c09c6b-33dd-4de9-aab8-222a50919654/whereabouts-cni/0.log"
Sep 17 16:53:37.274807 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.274776    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x5njn_34afb8a0-7a2b-4aa8-9eae-d11acf59fe8c/kube-multus/0.log"
Sep 17 16:53:37.450440 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.450372    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xh8xc_afc9f293-dabb-4bff-b819-41dc8625ce82/network-metrics-daemon/0.log"
Sep 17 16:53:37.474168 ip-10-0-142-18 kubenswrapper[2585]: I0917 16:53:37.474138    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xh8xc_afc9f293-dabb-4bff-b819-41dc8625ce82/kube-rbac-proxy/0.log"